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ABSTRACT 


The purpose of this study was to determine the most 
appropriate and effective manner in which to provide feed- 
back to the unit commander after a Marine Corps Combat Readi- 
ness Evaluation System (MCCRES). To this end research was 
conducted into the areas of management control and evaluation 
theory. 

The approach of the study was multi-disciplinary with a 
detailed emphasis on the economic question of how better 
distribution could be made of the scarce resources of man- 
power, equipment, and training time based on the results of 
MCCRES. To answer the questions a detailed field study was 
conducted in which interviews were completed with 37 Marine 
officers of rank from Captain to Colonel who had involvement 
with MCCRES as key billet holders. 

The results of the study are displayed as an eight-step 
feedback model which is based on accepted theory inthe fields 
of management control and evaluation theory. These results 
are communicated to the reader as a descriptive model. 

The incorporation of this model as the Standard MCCRES 
Feedback Procedure will enhance the value of the results 
to the evaluated unit and will improve the understanding of 


the resource allocation needs at all levels. 
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tee NODC ION 


The Military has long recognized the need for forces to 
be prepared for combat. The United States Marine Corps has 
historically been considered this country's force in readi- 
ness. A means to evaluate and measure readiness, called the 
Marine Corps Combat Readiness Evaluation System (MCCRES), was 
implemented by the Service in 1977. MCCRES was designed to 
provide readiness information to all levels of command. How- 
ever, the system's ultimate value lies in the communication of 
information to the unit evaluated and to the higher level com- 
mands that can support improved allocation of resources and, 
ultimately, a higher level of readiness. This research views 
the MCCRES from an interdisciplinary perspective and specifically 
looks at the economic consideration of improving the allocation 


of scarce resources of manpower, equipment and training time. 


A. PURPOSE 

The purpose of this thesis is to look specifically at the 
feedback resulting from a MCCRES evaluation and to consider 
all of the MCCRES information available to the Marine Corps 
and to determine how that information could be used to improve 


combat readiness. 


B. SCOPE/METHODOLOGY 
Initially a detailed literature search was conducted into 


the areas of management control and evaluation theory. 
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Following the detailed literature search, the background and 
development of MCCRES was reviewed. The MCCRES is applied 
throughout the Marine Corps to evaluate ground and air combat, 
as well as combat service support units. Most commonly the 
MECRES is applied at the infantry battalion and aviation 
squadron level. In total over 500 MCCRES's have been conduc- 
ted since implementation in 1977. Therefore, a sizeable body 
of Marine Corps Officers still on active duty have served in 
key billets during MCCRES evaluations. To capitalize upon 
their experience a field study was conducted which included 
interviews with the initial developers of MCCRES (both Marine 
and civilian personnel). In addition key questions on MCCRES 
feedback were posed to those who had been involved at all levels 
of MCCRES but with special emphasis on those who had served as 
unit commanders, senior evaluators, or division or squadron 
MCCRES officers. Based on this research the answers to ques- 


tions posed by this thesis were answered. 


©. ANALYSIS 

The information obtained was subjected to two levels of 
analysis. First specific responses to the field study were 
analyzed through the application of content analysis. Next 
an analysis of general responses to the data collected was 
conducted identifying MCCRES elements which were repeatedly 
presented by those interviewed. Finally, a summary of unsolic- 


ited comments were compiled to provide completeness to the 


i 





study and are suggested as areas for further research on the 


subject. 


D. CONCLUSIONS AND RESULTS 
The conclusions and results are provided in the form of a 

feedback model consisting of the following eight elements: 

1. Pre-MCCRES Brief 

2. Realtime Feedback 

3. Evaluator Worksheets 

4, Two-way Debrief 

a) Computerized MCCRES Printout 

6. Comparative Percentile Results 

7. Follow-up Report 

8. Trend Analysis 

Overall the MCCRES is viewed as a valuable and necessary 

measure of unit readiness. However, to maximize the effec- 
tiveness of MCCRES timely distribution of results must be made 
to the unit evaluated and all others directly in the chain of 
command. The timely distribution of results will allow for 
improved resource allocation. Specifically the distribution 
of results has a high value for improving the use of limited 
training time and the development of doctrine that supports 
the effective and efficient use of Marine Corps resources. 
Thus regular distribution of generic results should be provided 
to all training and doctrine commands throughout the Marine 


Corps to improve areas consistently identified as deficient. 
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It. BACKGROUND 


The Marine Corps is interested in assessing Unit Combat 
Readiness. Readiness is generally viewed as the peacetime 
level of preparedness to go into combat. The Marine Corps 
Combat Readiness Evaluation System (MCCRES) is designed to 
measure units' level of readiness through simulated combat 
conditions as determined by specific criteria set forth in the 
form of mission performance standards, tasks and requirements, 
and as judged by experienced, expert evaluators [Ref. l]. 

The readiness demonstrated is actually a complex set of re- 
sults based on the effective allocation of resources by the 
service and the unit commander [Ref. 1]. Management control 
encompasses all management actions taken by the service and 
the unit's commander to bring the unit to peak efficiency. 
The evaluation system employed is a detailed effort to quan- 
titatively determine the success of that resource allocation. 
This chapter explores the key concept of readiness as it has 
historically evolved. It discusses the basics of MCCRES and 
lays groundwork for the detailed research that is directed at 
answering key questions involving feedback of MCCRES data to 
the unit commander for the specific purpose of upgrading 


deficiencies. 


A. MARINE CORPS COMBAT READINESS EVALUATION SYSTEM (MCCRES) 
Historically the Marine Corps has been called upon as a 


ready force to meet contingencies on short notice world-wide. 
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The specific Marine Corps Missions are detailed in the National 
Security Act of 1947. The unique air-ground combat team organi- 
zation in which each unit ranging in size from the Battalion- 
Landing Team (BLT) to Marine Amphibious Force (MAF) is task 
organized with Marine Corps Air, Ground, Combat Support, and 
Combat Service Support elements making the organization uniquely 
Suited to respond rapidly to U.S. global needs. Recent inci- 
dents such as the 1979 Iranian Crisis and the 1982 Lebanese 
Crisis have placed current emphasis on this historic role. 
i) our pose 
The operational capabilities of combat units describe 

their abilities to function for their intended purpose. It 
is, therefore, necessary that some means be used to estimate 
their ability to perform their intended mission short of actual 
combat. In 1976 the Commandant of the Marine Corps (CMC) iden- 
tified the need to develop an operational readiness evaluation 
program. In his Posture Statement to Congress for 1978, shortly 
before MCCRES was formally introduced, General Louis H. Wilson, 
Commandant of the Marine Corps, made the following statement 
describing a need for: 

-..an improved readiness evaluation system to, provide 

a timely and accurate evaluation of the readiness of the 

Fleet Marine Forces, including reserve units, to accom- 

plish assigned missions. [Ref. 1: p. I-A-1] 
The Marine Corps began using the system in July, 1978. Since 
that time the system has been used to test all Marine Corps 
Infantry Units, Fixed Wing Squadrons, Rotary Wing and Obser- 


vation Squadrons, Combat Support and Combat Service Support 
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element, both regular and reserve, at least once every 24 
months. In his FY-84 Posture Statement to Congress, General 
Robert H. Barrow (the Commandant of the Marine Corps) made 
the following comment about MCCRES. 

A key element in the readiness equation continues to 

be operational performance standards. Even before 

these units deploy, we evaluate their capabilities 

to accomplish their assigned missions through the 

rigorous Marine Corps Combat Readiness Evaluation Sys- 

tem. The System objectively evaluates the combat 


readiness of all active and reserve FMF (Fleet Marine 
Force) Units. [Ref. 2: p. 47] 


2. ineory 
The goal of MCCRES is to test the combat readiness of 
Marine Corps units. These tests have been developed to evalu- 
ate the five types of units mentioned previously. There are 
difficulties in applying a standard such as MCCRES. The real 
question that needs to be answered for the organization is: 
meeene: Unit do the job?” [Ref. 3: p. lj] Since the real 
answer can only be obtained under actual combat conditions, 
we must substitute simulated combat for actual combat. Thus 
the question that is answered is: "How close is the execution 
mo doctrine?" [Ref. 3: p. 1] Thus for MCCRES we test adherence 
to doctrine, under simulated combat conditions as a proxy for 
the real question. 
Bee cecoring/ Results 
The guidance for MCCRES is defined in detail by Marine 
Corps Order (MCO) 3501.2, Volumes I-X. This provides details 
on how the MCCRES is applied to each type of unit. The evalua- 
tion is broken down into ten (10) categories that are mutually 


exclusive and collectively exhaustive. 
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1. Reporting to higher level of command. 
Teese paring £0r Operations 
3. Communicating (including communications SOP) 


4. Performing as Marines (discipline, dispersion, 
camouflage, concealment, using weapons, and so on) 


ee Delivering supporting fire 
See Lanning Of Operations 

fe Conforming to doctrine 

8. Executing Operations 


9. Providing combat service support (including medical 
Support) 


ie supervising required actions by individual Marines. 
Each category corresponds to a vital aspect of the unit's 
performance [Ref. 3]. 

For an Infantry Battalion the evaluation is divided 
into 19 Mission Performance Standards, 145 Tasks, and 793 Re- 
@merements. Each Battalion is not tested on all MPS, Tasks, 
and Requirements [Refs. 1, 4]. The final result is a "Combat 
Ready/Not Combat Ready" for the unit evaluated based on the 
overall judgment of the senior evaluator. Results are also 
provided as percentile scores for the Section, MPS, and Tasks. 

The format for information provided is as follows: 


infantry Una 


Section Ze 0 Operations Performed % Score 
MPS ed. Actions by Marines % Score 
Task 2.A.1 Discipline % Score 
Requirement Diese cleae ts self Discipline %$ Score* 


Bete: Oniy a yes/no is given for a Requirement, thus 
100% = Yes and 0% = No for each Requirement Graded 
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For example, the Infantry Evaluation 1s subdivided 
into four Sections: Section 2.A identifies standards applica- 
ble to all evaluations. It is divided into three MPS, the 
first of which is Action by Marines. ‘Actions by Marines' is 
subdivided into 13 Tasks--the first, ‘Tasks, Discipline' is 
subdivided into nine Requirements. The first of these require- 
ments is 'Self-Discipline.' Requirements are evaluated as: 
Yes/No/Not Applicable. Section, MPS, Tasks, and Requirements' 
overall evaluations are each given a weighted percentile score 
based on the number of Yes's for Requirements under the respec- 
tive category. 

4. Procedure 

The MCCRES is divided for simplicity into five phases. 
Phase One requires determining the appropriate MPS to be evalu- 
ated based on Marine Corps Order 3501.2, Vol. I, and the appro- 
priate volume that applies to the unit being evaluated--for 
instance, air or combat-Service support. Phase Two consists 
of briefing evaluators [Ref. 5]. Phase Three is the actual 
evaluation under simulated combat. During this phase the 
evaluators actually make the Yes/No evaluations of the require- 
ments. They are guided by detailed Key Indicators (KI) which 
assist the evaluator in making the appropriate choice [Ref. 4]. 
Phase Four consists of compiling the evaluation results and 
determining the percentile scores as well as the "Combat 
Ready/Not Combat Ready" determination by the senior evaluator. 


Officially, there is no relationship between the numeri- 
cal score for the Battalion and the Combat Ready/Not 


cS 





@embat Reacy rating. It 1S possible, for example, 

EiaeOne Hattalton with an overall numerical score of 50 

could be judged Combat Ready, while another having a 

numerical score of 75 could be judged Not Combat Ready. 

Per. 6: p. 40] 
Phase Five, the final phase, is the debrief of the evaluated 
mame by the evaluators as well as the detailed written report 
(computer printout) which is provided to the evaluated unit. 
Other reports are forwarded to Headquarters Marine Corps (CMC)-- 


an initial message report within 10 days and the detailed re- 


port within 30 working days [Ref. 1] 


Bee READINESS 
At the center of any Readiness Evaluation System is under- 
standing what 1s meant by the key concept of "readiness." The 
inherent difficulty in describing this term makes any evalua- 
tion, measure, or model of readiness all the more complex. 
Thus we must explore the meaning of "readiness." 
i, Definition 
Usage of the term "readiness" has tended to change 
even at the highest levels within the Department of Defense 
(DOD). In FY 1977, then Secretary of Defense, Donald H. 
Rumsfeld, used the following definition of readiness: 
"Readiness" is a concept that integrates the diverse 
mactOrs that afrect the ability to deploy, engage, and 
Stistain effective combat forces. It starts with the 
meerall ability and proficiency of U.S. fighting men... 
An equally important determinant of overall readiness 
ms tne availability, capability, and condition of the 
morce’s fighting equipment. [Ref. 7: p. 2] 


In FY 1978 the same Secretary of Defense is quoted 


as follows: 
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"Readiness" refers to the capability to respond adequately 
to diverse situations and to sustain that response as 

long as necessary. The "Readiness" of Defense Combat 
Forces depends on a myriad of diverse and often inter- 
related factors. [Ref. 7: p. 2] 

In recent years the concept has taken a more stan- 
dardized definition under the current administration. How- 
en, in reviewing historical literature concerning readiness, 
it must be remembered that it does not have consistent meaning. 
In address to Congress on February 8, 1982, and January 3l, 
1983, the Secretary of Defense Weinberger used the following 
definition: 

Readiness is the ability of a forces, units, weapons 
- systems, or equipment to deliver the outputs for which 
they were designed (including the ability to deploy and 
employ without unacceptable delays). It depends on 
having the required quantities of equipment in the hands 
of the units on a day-to-day basis and on having the 
required number of adequately trained people assigned 
with the necessary mix of grades and experience level and 
to ensure that people and machines can work together. 
[Ref. 8: p. 1-28] 
ahs Gefinition includes training (individual and unit), 
material, equipment, logistics, and personnel all as part of 
the readiness definition. 

For the purpose of continuity and brevity, the follow- 
ing definition of readiness is used on a day-to-day basis by 
the Marine Corps and is the accepted definition from JCS, 
Publication 6, used for the DoD Unit Reporting System 
(UNITREP). 

Readiness: ability of forces, units, weapons systems, 
or equipments to deliver outputs for which they were 


designed (includes the ability to deploy and employ 
without unacceptable delays. [Ref. 9: p. 1] 


a 
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It is further amplified as follows: 

Readiness is essentially a measure of pre-D-Day status 

(extending at most into initial combat operations) 

while sustainability is a Post-D-Day measure. Hence, 

we often speak of peace time readiness, but combat 

Sustainability. [Ref. 9: p. l 
For consistency this definition of readiness, as amplified, 
will be used throughout this study, since it provides the 
currently accepted DoD definition. In the literature the 
terms 'effectiveness' and 'sustainability' are often inter- 
changed to indicate Post-D-Day combat capabilities. Because 
of its broader implications the term effectiveness will be 
used herein to mean Post-D-Day status. Hereafter we can refer 
to Pre-Combat Readiness and Combat Effectiveness. 

2. Indexing 
The difficulty in defining readiness may be exceeded 

only by the difficulty of measuring it. Many attempts have 
been made to index readiness, but to date none have been suc- 
cessful or achieved wide acceptance. As described by Barzily, 
Marlow, and Zacks, in their "Survey of Approaches to Readiness," 
the motivating factor common to all attempts is to measure 
the effects of budgetary changes on readiness. They reviewed 
a number of readiness indexes, such as the U.S. Navy METRI 
Project, MARIS Project, MAXCAP Models, and others and found 
that: 

The readiness indexes posed do not attain the desired 

objective. They are usually very insensitive to changes 

that occur at the lower echelons and, furthermore, they 


are generally improper indexes of readiness. [Ref. 10: 
Pe 25) 
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3. Economic Question 

The context of readiness described in both Definitions 
and Indexing has strong economic overtones. The approach 
herein will be to treat readiness as an economic question, 
1.e., the allocation of scarce resources of training time, 
material, equipment, logistics, and personnel as inputs with 
the desired results of maximizing outputs. The true output of 
a combat organization is effectiveness in combat. There is 
obvious difficulty in measuring that output. Because of this 
difficulty we generally turn our attention to proxy measures 
Such as number of aircraft or number of ships. The difficulty 
of this problem was recently stated as follows in a study by 
Rand for the Assistant Secretary of Defense/Manpower, Reserve 
Affairs, and Logistics: 

The output of an Army Maneuver/Firepower Unit is 
especially difficult because such units have no single 
Sureut Or product that can be directly related to 
Mecar Mission. {[Ref. 12: p. 7] 

In a recent study by Steven L. Funk, several observa- 
tions were made which have application here. He states that 
although peacetime Army readiness is an honored theme, it is 
not always practical. This is a result of preparedness not 
being the actual primary peacetime mission, but just one of 
several competing missions that take resources of manpower, 
Saeepment, and training time. Funk cites major factors that 
detract from the unit being able to train for combat situations: 

(1) Lack of resources to accomplish requirements 


(i1.e., skilled personnel, time, ammunition, 
repair parts, and facilities), 
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(2) changing and competing priorities that fragment 
resources and negate planning, and 
(3) a climate that deprives unit leadership of 

[eer On maneivermsoace and Enagments their 

attention (i.e., centralized decision making/ 

control, burdensome administration, low unit 

cohesion and ambiguous institutional values. 

[Ref. ll: p. 6] 

Although the previous study was made of the U.S. Army, 
many of the detractors apply equally to the Marine Corps. 
Some policies such as Unit Rotation (i.e., the policy of main- 
taining unit integrity by replacing an entire battalion/ 
squadron rather than replacing individuals) and Deployment- 
Lock-On periods (i.e., the policy of asSigning resources to 
a unit well in advance of a planned deployment and then main- 
taining and replacing those resources through the training 
cycle and into deployment) have lessened the negative effects. 
However, in the Marine Corps the problem of competing priori- 
ties will always exist. The unit commander is constrained 
by many elements beyond his control [Ref. ll]. Thus the ques- 
tion of unit readiness is certainly one of economic priorities 
or competing priorities. A large part of the readiness results 
then depends on the effective resource allocation both to the 
unit and within the unit. Thus MCCRES attempts to evaluate 
the readiness of a unit through simulated combat conditions 


and to measure how closely the unit performance conforms to 


doctrine. 


C. MCCRES AS THE EVALUATION SYSTEM 
1. Research 
The evaluation system employed by the Marine Corps 


has undergone much study from the standpoint of developing an 
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objective versus a subjective system of evaluation. Much of 
the statistical work for this has been done by the George 
Washington School of Engineering and Applied Science [Refs. 3, 
4}. Additionally, the question of evaluator bias was recently 
studied at the Naval Postgraduate School, Department of Adminis- 
trative Science [Ref. 5]. 
As a result of the George Washington University studies, 
it was determined MCCRES had the following merits: 
a. Most requirements are requests for descriptive data 
and not for judgments. Judgments were previously 
made by defining the requirements and assigning 
weights. 
b. The details of the doctrine are given and interpreted 
in the requirements, thus avoiding the possibility 
of being mis-interpreted or forgotten by the 
evaluators. 
c. The execution of most requirements consumes sHort 
time periods and thus the evaluators' memories are 


not overburdened. 


d. Assigning a score of a YES or NO is easier than 
asSigning scores on any other scale. 


e. The set of the requirements exhausts the details 
Meee weweemrions, (Ret. 33. p. 25) 


Though the George Washington University research identifies 
merits of MCCRES, it does not address the question of what 
feedback should be provided to the evaluated unit. 
2. Feedback 
The purpose of this research was to determine the types 
Of feedback that are most useful and valuable to an evaluated 
unit. Feedback is a necessary part of any control or evalua- 


tion system. In December, 1981, B. General A. A. Sordo, then 
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Director of the Marine Corps Training Division, made the 
following comment: 
The evaluation and feedback process in training is 
one where we need to do substantial work. MCCRES 
helps us greatly. In the future, as we link the 
individual training standards of our ITS System to 
MCCRES Mission Performance Standards, we'll be able 
to do better! But above and beyond that, our 
measurements of adequacy and effectiveness are for 
the most part decentralized and highly subjective 
and impressionistic. In the long run we rely on 
our assessment of the unit or organizational capabili- 
ties as the best gauge of adequacy in training 
preparation. [Ref. 13: p. 54] 
D. RESEARCH QUESTIONS 
As a result of the need to define specifically the feed- 
back and information needed by evaluated units as the result 
of MCCRES, this study was undertaken. The following questions 
are asked by this thesis: 
ime Primary 
a. After a MCCRES, what information should be fed 
back to the evaluated unit commander? 
b. In what form should the evaluation feedback be? 
c. What channel(s) should be used to provide the 
feedback information to the unit commander? 
2. Secondary 
In addition to the specific questions above, there are 
several subsidiary questions which arise as a logical conse- 
quence of the primary questions. They involve the following 


topics: (1) validity of comparing MCCRES results given under 


differing conditions, (2) MCCRES contribution to effective 
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training, and (3) time availability and followup to ensure 


correction of deficiencies identified during the MCCRES. 


BE. METHODOLOGY OF LITERATURE RESEARCH 

To properly answer the questions posed, the author took 
a multi-disciplinary approach. The theories of management 
control and evaluation provide a great deal of insight toward 
understanding the complexities of the information needs. Thus 
these two fields will be explored in detail in the next chapter. 
In addition much work has been done in the area of modeling 
effectiveness. This has been done from the perspective of 
applied, conceptual, statistical and economic models. 
Since these provide rich theories for the understanding of 
the measurement and indexing of effectiveness, these concepts 


are explored. 


F. SUMMARY 

This chapter was designed to provide a history of MCCRES, 
establish the need for such a system from the Marine Corps 
perspective, give some basic understanding of the system, and 
provide a detailed understanding of the inherent difficulties 
of efforts to evaluate the concept of readiness. 

Readiness is approached from the standpoint of economics, 
that is the application of scarce resources (inputs) of time, 
men, materials to the process of simulated combat. The unit 
commander controls the organization with a highly sophisticated 


Management control system in accordance with the external and 
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internal objectives and policies. MCCRES is a unit evaluation 
Pestem which evaluates the management control process of that 
organization. The primary questions are then presented as 

the basis of this research. The following chapters will ex- 
meore Dbrietly the theory of management control systems and 

in detail evaluation theories and models with specific empha- 


sis on feedback mechanisms. 
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III. MANAGEMENT, MANAGEMENT CONTROL, EVALUATION, 
AND FEEDBACK 

The topics to be covered in this chapter are as follows: 
first the topic of management will be described in a general 
context; management control and its implications for feedback 
will be explored; then, a series of selected evaluation 
theories will be presented in detail. Finally a series of 
evaluation models with application to the military will be 
provided. Emphasis in these models will be on feedback 
theory and the ideas on feedback posed by the authors. 
A summary of feedback as developed by the models will be pre- 


sented also. 


A. MANAGEMENT 

Management is described as: (1)"art or act of managing; 
conduct; control; direction. (2) Judicious use of means to 
accomplish an end; skillful treatment" [Ref. 14]. Webber 
describes it in many contexts--experience, training, practice, 
theory, art, and science [Ref. 15]. The simplest definition 
and one often heard is the art of getting things done through 
people. The elements, however, that all descriptions of 
Management have in common are that managers engage in two 
important activities: planning and control [Ref. 16]. It is 
important to clearly make the distinction between the terms 


Management and leadership. The military has long recognized 
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that the term "leader" has much broader implications than that 
of manager. The unit is more than the sum of its parts; and, 
therefore, the military commander's responsibility is not only 
for the technical and tactical proficiency of the unit, but 
for the overall cohesive functioning of the organization [Ref. 
17]. In this context it is fair to say that management skills 
are a necessary subset of leadership and that the good leader 
1s necessarily a good manager, however, the converse of that 


statement is not necessarily true. 


B. MANAGEMENT CONTROL 
Robert Anthony placed management control in a hierarchical 


framework as follows in Figure 3.1. 


Operational Control 





Figure 3.1. Anthony Framework 


He described this classification for any large organization 


where these functions are present. 
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Strategic planning is the process of deciding on 
objectives of the organization, on changes in the 
objectives, on resources used to attain these 
objectives, and on the policies that are to govern 
the acquisition, use, and disposition of these 
resources. [Ref. 18: p. 16] 

He further described management control as follows: 
Management control is the process by which managers 
assure that resources are obtained and used effec- 
tively and efficiently in the accomplishment of the 
Organization's objectives. {Ref. 18: p. 17] 

Strategic planning is the highest level of management 
where the long. range objectives, policies, and plans of the 
Organization are made. The management control level involves 
middle management, the timeframe is shorter, about 12 months, 
and the planning involves budgeting, training, etc. The lowest 
level of control is operational where the day-to-day function- 
ing of the organization is carried on; the implementation of 
plans and policies are executed at this level. 

Lebas describes management control as an enlarged feedback 
loop which affects human behavior and performance [Ref. 19]. 
Hofstede also provides a useful definition of management con- 
erol "...aS a pragmatic concern for results, obtained through 
people" [Ref. 24: p. 193]. 

Given the purpose of this study is to identify feedback 
needed for improved resource allocation, it is important to 
focus on the terms "effectiveness" and "efficiency", which are 
present in many of the definitions of management control. In 


this context the terms effectiveness and efficiency are defined 


as follows: 
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Effectiveness...relates to the accomplishment of the 
cooperative purpose...when a specific desired end 
is attained we shall say that the action is effective. 
beet. 18: p. 27] 
Efficiency...refers to the engineering sense of... 
the optimum relationship between given inputs and 
@utputs. {Ref. 18: p. 27] 
In Lebas' work he identified the major schools of thought 
on management control. 
Ii. Organizational Process Approach 
The Organizational Process Approach is that taken by 
Anthony. He divides the feedback loop into three distinct 
parts, described previously as strategic planning and manage- 
ment control and operational control. This approach reduces 
the human motivational problem to what Anthony calls "goal 
congruence" or the various ways in which the manager can be 
encouraged to take actions which are in the best interest of 
the company. The goal congruence is then reduced to procedures 
within the organization such as performance evaluation, report- 
ing, and budgeting [Ref. 19]. 
2. Informational Economics Approach 
The Informational Economics Approach is a more abstract 
theory. It proposes that only models based on expected utility 
maximization need to be considered. A set of subjective 
probabilities are assumed to be known by the decision-maker 
and that the decision-maker will always act to maximize utility. 
In this approach, good performance is described as the action 


which minimizes the end result differences between the expected 


outcomes and the actual outcomes. The emphasis, however, is 
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on the decision model not the outcome. A further aspect of 
the theory is its emphasis on maximizing the expected utility. 
This leads to the conclusion that the control process is a 
prior phenomenon that suggests management spend more time 
preparing decisions than evaluating outcomes [Ref. 19]. 

The major limitation to the Information Economics Ap- 
proach is that it is all based on the Savage-Rational Man 
Model and that an effort will be made to maximize utility in 
every case. Although an interesting theoretical model, it 
relies heavily on the calculus of maximizing utility in each 
and every case [Ref. 19]. 

3. Behavioral Approach 

This is actually a loosely structured set of approaches 
to the concept of management control taking ideas from many 
theories. Two descriptive ideas that it entails from Herbert 
Simon are: 


Satisficing...interrupting the action selection process 
as soon as the first acceptable action has been found. 


And 


Peunded Reality...the notion that man is a plecewise 
factional. [Ref. 19] 


Other ideas presented by Ouchi and referred to by 
Lebas have shown that: 


»~»e-rigorous output control is the most effective way 
to induce good performance in managers. [Ref. 19] 


Other topics in the behavioral approach deal with the 
structural and inter-personal variables in management control. 


The power, therefore, of each structural center lies in its 
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femeity tO reaquce uncertainty for other units. The inter- 
personal effects on behavior are viewed by Lebas in terms of 
Kantz and Kahn's model of role sender and role receiver. Here 
the control will be good if the expectations are interpreted 
accurately by the receiver and if the single receiver is not 
subjected to conflicts in role expectations in the different 
social systems in which it operates. Additionally, control 
Requires acceptance of the role by the receiver [Ref. 19]. 

Lebas views the real value of behavioral approaches 
in their diversity and being based on observations of people 
in real situations. The limitations come from their multi- 
plicity and that creates difficulty in setting any sort of 
coherent guidelines for a manager to employ [Ref. 19]. 

4. Integrated Approach 

The previous broad structures of management control 
system have been integrated by Ansari in what he refers to 
aS an operational systems concept [Ref. 20]. He views the 
previous perspectives on management control systems in two 
versus three broad categories. Grouping the structural view 
(Operational Process and Informational Economics) and the be- 
havioral views: 

a. Two Perspectives 

Ansari describes the structural view of management 

control as that adapted by researchers in cybernetics, account- 
ing, and management information systems--those that concentrate 


primarily on the information and communications aspects of 
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control systems. The behavioral approaches are identified as 
those that are based primarily on human behavior in organi- 
zations and regard control as a problem of encouraging sub- 
ordinates to achieve performance goals. In this context he 
uses control system to describe: 

...those arrangements and actions designated to 

facilitate its members to achieve higher performance 

with least unintended consequences. [Ref. 20: p. 102] 

b. Elements of Management Control Systems 
There iS agreement among most authors reviewed 
from both the structural and behavioral schools that all 
management control systems consist of two elements. The 
first being "an information network which prescribes the rules 
Bem measurement, collection, processing, and transmission of 
information" [Ref. 20: p. 102}. This element causes the 
information on performance, goals, outputs, and exceptions 
from plans to be transmitted to managers. The second element 
is the set of social relationships through which the control 
System achieves the organizational goals [Ref. 20]. 
c. Joint Consideration 
Ansari argues that management control is actually 

best approached from a combination of the structural and be- 
havioral ideas. He declares that the current phase of design 
1S more situational and that it focuses on the more important 
issue of improving performance instead of the narrow concept 
of constraining behavior. The information that is provided 


passes through two phases. First, it must be perceived before 
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it can be used and second not only managers, but subordinates, 
must also use information for self-evaluation and guidance of 
performance. He further identifies five considerations which 
are characteristic of all information structures. 
1. Nature of Measures--the subordinates" motivation 
is affected by the completeness, objectivity, 
and influence of the characteristics of perfor- 
Mance measures. 
2. Source and Order of Presentation--The source of 
the information must be credible and the order 
of presentation can change the perceptions of 
the information. 
3. Timing--Both the speed and frequency are impor- 
tant, too long an interval may cause the user 
to lose interest or be distracted. 
4. Route--The route may alter the information and 
thus the sender and receiver may not be sharing 
the same information. 
5. Shared Information--The others with whom the 
information is shared thus may affect the recipi- 
ent's view of the information with regard to 
fairness and accuracy. [Ref. 20] 

All of these considerations must be made by the 
designer in any information, evaluation system. To summarize 
the characteristics of the integrated model, this takes into 
account both the structure of the organization and its infor- 
mational systems as well as the social side considering subor- 
dinate personality and leadership style. The designer of a 
Management control system should combine components (such as 
an evaluation system) in such a way as to minimize cognitive 
conflicts and encourage behavior which resolves conflict with 


positive results for the organization. It should also be kept 


in mind that if rewards for performance are too highly 
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Seneingent On OUEpPUtTS of the information structure, the greater 
the chance that there will be controversy over the output 
[Ref. 20]. A good example of this controversy is discussed 
by Anthony Hopwood in the way in which accounting data was 
used in the performance evaluation of managers. 
Based upon a study he conducted, Hopwood identi- 
fied four ways in which accounting data was used in evaluation. 
(1) Budget Constrained Style--required to meet a budget 
but not concerned about costs. 
(2) Budget Profit Style--concerned with both meeting a 
budget and with costs. 
(3) Profit Conscious Style--concerned with costs but not 
with meeting a budget. 
(4) Non-Accounting Style--not concerned with meeting a 
budget or with costs. [Ref. 21] 
He concluded that based on management style, many dysfunc- 
tional behaviors may occur as the result of emphasis on 
accounting data results in evaluations [Ref. 21]. 
5. Other Thoughts on Management Control 
Management control contains many rich concepts which 
play a great part in the theory of evaluations. San Miguel 
has made several interesting observations on management 
eomerol : 
The end is a system that enable managers to make sound 
decisions as to the efficient and effective allocation 


of human, physical, and financial resources to attain 
the objectives of the organization. [Ref. 22: p. 177] 
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San Miguel further characterizes management science's 
attempts to deal with modeling of complex control systems. 
He characterizes the quantitative and economic decision tools' 
attempt to simplify the decision making in large scale organi- 
zations as falling short of their objectives--primarily be- 
cause of the lack of knowledge of human behavior and the 
ability to quantify it. He further cites measurement and 
communications to be at the center of control systems, internal 
planning, and reporting. There is general agreement in the 
Management control literature that measurement and communica- 
tions systems have an important impact on the behavior of 
individuals in organizations, their motivations and, thus, 
their performance. Therefore, the areas of measurement and 
evaluation are legitimate concerns for the design of any 
Management control system [Ref. 22]. 

6. Summary 

The intent in this section was to set the stage through 
the discussion of the broader topics, management and manage- 
ment control, for what is to be developed in the next section 
On evaluation. Management control is a complex topic which 
encompasses the broad aspects of all the control systems im- 
posed on the organization through strategic plans and policies 
as well as those developed within to assist the manager in 


Meese cOntrol internally. 


eC... EVALUATION OVERVIEW 
Evaluation models that are directed at readiness all 


embody a common goal. Each in its own particular way attempts 
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to identify a definitive measure or some particular relative 
measure of the unit's Pre-D-Day preparedness for combat. 
Preparedness may be viewed as readiness which acts as a proxy 
Measure for the military unit's ability to perform in combat. 
Similar efforts are made to evaluate the ability to perform 
Mmmeeivilian organizgations. In its simplest form, output is 
measured by production level or quantity produced by a produc- 
tion facility in a given time period. Various attempts are 
also made to measure profitability of the firm. Microeconomic 
theory assumes that each firm is attempting to maximize profit 
[Ref. 23]. Therefore, some measure(s) are applied to the 
fiscal results of each period to determine the financial per- 
formance of the organization during the period [Ref. 23]. 
fe tarly it can be assumed that each unit tries to maximize 
its me ainess.” This section reviews efforts that have been 
made to evaluate, measure, and model performance both in 
theory and practice. This section covers the general theory 
of evaluation, and looks at several well-known evaluation 
=syocems, 
1. Evaluation Defined 

The term evaluation is defined differently by various 
authors; however, Stufflebeam brings together three generally 
accepted definitions of evaluation and provides a comparison. 

a. Measurement Definition 

Evaluation is identical to measurement. It builds 

Sm@eatctcempts to measure psychological attributes or character- 


istics [Ref. 24: pp. 9-10]. 
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b. Congruence Definition 

Evaluation is a comparison of the congruence 
between performance and objectives. This provides emphasis 
on objective performance also [Ref. 24]. 

The measurement definition considered evaluation 
to occur after the fact and to measure some attribute. In 
contrast the congruence definition refers to on-going evalua- 
tion in meeting the objectives [Ref. 27]. According to 
Stufflebeam organizations will pick objectives which are 
specific and have objective measureable results. That is be- 
Cause to make the congruence definition work, there must be 
some selection of objectives to be measured. Stufflebeam 
further indicates that the congruence definition creates a 
need for the evaluator to find short term measures of per- 
formance. These indicators, identifiable outputs, or behaviors 
come to be viewed as performance. These identifiable outputs 
become the ultimate criteria of all organizational decisions. 
The congruence definition has advantages, but the need to 
find measureable objectives creates problems [Ref. 24]. 

c. Judgment Definition 

The professional judgment definition allows full 
evaluation of all organizational attributes both quantifiable 
and non-quantifiable, and is easy to implement [Ref. 24]. 
This definition is often applied where the dimensions to be 
measured are difficult to define and more difficult to quan- 


tify. The judgment definition has its problems. The evaluation 
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is, by definition, based upon the judgment of an individual 
which lacks objectivity identifying what data are used and 
how they are used [Ref. 24]. 
Table (3.1), taken from Stufflebeam, provides a 
comparison of advantages and disadvantages of each definition. 
wee ec inciples or svaluation 

There is a good deal of general agreement that for 
an evaluation to be effective it must be consistent with the 
purposes, objectives, and goals of the activity being 
evaluated [Refs. 5,25,26]. 

a. Evaluations must be conducted in terms of purpose-- 
the evaluator and the evaluated must fully participate 
and work for the common goal. 

b. Evaluation must be cooperative--all involved as both 
the evaluators and evaluated must fully participate 
and work for the common goal. 

c. Evaluation must be continuous--it must be on-going; 

a one time effort with no followup is an affront to 
the professional concepts. 

d. Evaluation must be specific--specificity is the key, 
generalizations do little good to help remedy defi- 
ciencies or to identify true strengths. 

e. Evaluation must provide the means and focus for 
trainers to be able to appraise themselves, their 


pHeactileces, and’ their products. 


f. Evaluation must be on a uniform and objective methods 
and standards. 


3. summary 
The definition and principles of evaluation were pre- 
sented in this section. The evaluation definition as described 
by Stufflebeam has three distinct interpretations. There is 


more general agreement on the principles of evaluation and 
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(1) 


(2) 


(3) 


mabe: 3 i 


Advantages and Disadvantages Accruing from Different 
Definitions of Evaluation 


Measurement 


Congruence 


Judgment 


Source: 


Adopted from Ref. 


Advantages 


Builds directly on scien- 
tific measurement 
Objective 

Reliable 


Data mathematically 
manipulatable 


Norms and standards 
emerge 


High degree of integra- 
tion with the process 


Data available on 
process and structure 


Possibility of feedback 


Easy to implement 


Brings all variables 
into consideration 


Takes experience and 
expertise into account 


No time lag while wait- 
ing for data analysis 


24: 


Pe. 
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Disadvantages 


Narrow instrumental focus 


Inflexible because of time 
and cost to produce new 
instruments 

QObscures judgments and the 
criteria for making then 


Eliminates variables cur- 
rently considered as not 
measurable or labels them 
unimportant 


Places evaluator in technical 
role 
Focuses narrowly on objectives 


Elevates behavior as the 
ultimate criterion of every 
action 

Focuses on evaluation as a 
terminal process 


Dictated mainly because 
of ignorance or lack of 
sophistication 


Questionable reliability 
Questionable objectivity 
Not susceptible to ordinary 


scientific, prudential 
measures 


Both data and criteria are 
ambiguous 
Generalization very difficult 
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to a large degree these embody a common sense approach for the 
usefulness of any evaluation. Next, four evaluation theories 


will be presented. 


D. EVALUATION THEORY 

There are many theories of evaluation. There is no attempt 
herein to be all inclusive. What is intended is to introduce 
a variety of theories which exist and to show their relation- 
ship to the questions pursued in this research. 

1. Management by Objective (MBO) 

In recent years probably no system of personnel manage- 
ment has gained more attention than Management by Objective 
introduced by Peter Drucker [Ref. 27]. The MBO approach is 
applied to individual as well as organizational evaluation sys- 
ietitS « 

The MBO Approach assumes that one would perform more 
effectively because you have planned your own objectives 
geeancould control your own behavior. [Ref. 15: p. 316] 

MBO takes a humanistic or human values approach to 


managing people. The basic steps in MBO are as follows: 


1. The subordinate proposes goals for the next time 
Berl od,. 


Z@. the subordinate and superior discuss, modify, and 
reach agreement. 


3. Periodic formal and frequent informal review. 


4. Subordinate reports on performance at the end of 
the period. 


5. Repeat the cycle. [Ref. 15: p. 316] 
MBO deals well with a number of problem areas in evaluation 


systems. It is designed to create more two-way communication. 
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It has found success in both the private and public sector. 
In their article on "Employee Growth Through Performance 
Management," Beer and Ruhr found success with employment of 
MBO at Corning Glass Co. [Ref. 28]. In addition, the Depart- 
ment of Health, Education, and Welfare found it a most valuable 
system in pursuing management and control of their many diver- 
Sified programs [Ref. 29]. Figure 3.2 provides a simple model 
of how MBO is designed. It should be noted that the MBO model 
focuses on performance as identified and measured through ob- 
jectives agreed upon by supervisor and subordinate [Ref. 30]. 
Feedback in the MBO System is scheduled at regular 

intervals. The meetings can be used to compare results to 
the objectives set as well as to help make future objectives 
more realistic. The objectives themselves are determined by 
higher level management, but in coordination with the employee 
Or unit. To a great extent the comparison of goals set and 
achieved are reported by the individual being evaluated. 
Typically the manager submits an evaluation of each objective 
based on the following classification system: 

me oatisfactory 

2. Minor Problem 

3. Major Problem 
Feedback is handled by face to face conferences between the 
evaluator and employee. 

In a recent Marine Corps Gazette article, Major James 


M. Clarke reflected his good experiences with the MBO merit 
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Organizational 
Mission 


Establishment 
of Key 
Result Areas 


Supervisor's Subordinate's 






Tneuc: tO Pa SuiEsco 
Recommended Potential 
Objectives Objectives 


Possible Periodic 
Revision of Reviews of Counseling 
Objectives Progress 





Source: Adapted from [Ref. 30: p. 223] 


Figure 3.2. Sample MBO Model 
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pay system used with high level GS employees. He referred 
to MBO in the context of a personnel evaluation system; how- 
ever, it also has application here: 

To improve performance counseling, we must get away 

from subjective judgments through a more formal system 

which concentrates on objectives. The merit pay 

system provides a good model. [Ref. 31: p. 47] 

Major Clarke's article provided support for the MBO 
method because clear objectives are established and regular 
communications are encouraged. In the Marine Corps' personnel 
system, aS in other areas of inspection and evaluation, pro- 
fessional judgment has historically been used as the primary 
method of evaluation and lack of specific objectives has rein- 
forced the reliance on the judgmental method. 

MepCmoljces scmgmecatmemonasis On the objectives and the 
regular feedback on how the evaluated worker is measuring up 
to those objectives [Ref. 15]. MBO further emphasizes dis- 
playing the individual's strengths and weaknesses to himself, 
rather than to others. Although MBO emphasizes timely, accu- 
rate, and objective feedback, the details of that feedback 
can only be spelled out when a system is input to a specific 
Organization [Ref. 28]. It further encourages the feedback 
be given on a continual basis, not "saved up for an end of 
the year inquisition" [Ref. 15: p. 317]. 

2. Evaluation and Authority 

A comprehensive and ongoing study of evaluation and 

authority was conducted by Scott and Dornbusch (1977). In 


their study of professional organizations, results were 
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gathered from university faculty members, clergy, public school 
teachers, principals, nurses, and others. They found: 
Organizations are power structures in which some 
participants give differential access to organizational 
rewards and penalties in order to control other 
participants. [Rete 322.0. 134) 
The Scott-Dornbusch studies present a conceptual model 


of the relation of evaluation and authority. As a result of 


these studies, they developed the model presented in Figure 





So. 3. 
Nature of 
task 
Task assigned to C Task Weights 
properties and 
cs standards 
Performance 
Oe Values 
4 — a on properties |! 


Source: Adapted from [{Ref. 32: p. 20] 


Figure 3.3. Model of Evaluation Process with Communica- 
tion Links Among Right-Holders 


The model is a systems analysis approach which provides for 
monitoring and regulating by means of a feedback loop. This 
approach concentrates on the involvement of each actor in the 


process. The actors in the Scott-Dornbusch model are described 
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as follows: The allocator gives the responsibility for per- 
forming the task. The criteria setter is concerned with 
assessing the effectiveness and efficiency of a task per- 
formance. To a large measure the criteria is determined by 
the goal toward which the performance is directed. The 
determination of which information to use in order to arrive 
at a performance evaluation is the responsibility of the 
sampler. The appraiser uses the information on the sampled 
indicators and transforms the observed values into scores. 
The goal of the evaluation process is to provide a measure of 
the performer based on performance and outcome [Ref. 32]. 
The manager must get each of these individuals in the decision- 
making process and coordinate their activities. Emphasis is 
placed in this model on communications among the right-holders. 
The right-holders are those given authorized power or authority 
rights by the organization. The authority rights are granted 
to the allocator, criteria setter, sampler and appraiser. 
There are a variety of interconnecting links which may exist 
between these rights-holders which define the authority struc- 
ture within any given organization. Thus the formal evalua- 
tion process is based on the structure of the interconnection 
among the right-holders [Ref. 32]. 

The Evaluation and Authority Feedback Model describes 
performance by direct and indirect measures. Examples could 
be the number of units produced as a measure of labor produc- 


tivity (direct). An indirect measure is the number of professional 
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Papers written to evaluate the productivity of a scientist or 
scholar [Ref. 32]. The variety of samples requires judgment 
emetne part of the evaluator to determine what is high or low 
performance so judgment must be used by the evaluator. Thus, 
accurate appraisal of the sample taken requires a complete 
knowledge of the task, the performance, and the specific 
circumstances. Based on all of these measures the decision 
is then made on what, and if anything, should be conveyed to 
the performer concerning quality of the task performed (Ref. 
2). 
Pee 2 aaceicevor Program Evaluation 
Anderson and Ball take an applied approach to program 


evaluation. They identify six specific purposes of program 


evaluation [Ref. 33] 
1. to contribute to decision about vorogram installation, 


2. to contribute to decisions about program continuation, 
expansion, or certification, 


3. to contribute to decisions about program modifications, 
4. to obtain evidence to rally support for a program, 
5. to obtain evidence to rally opposition to a program, 


6. to contribute to the understanding of basic psychologi- 
cal, social, and other programs. [Ref. 33: p. 4] 


Anderson and Ball emphasize the practice aspect that 
evaluations can address a wide variety of questions and pro- 
vide many useful services. Their experience is primarily with 
social programs, but it is applicable to this study. They 


identify a total of seven types of evaluation methods: 
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1. The experimental and quasi-experimental study, 

2. Correlation methods, 

3. Surveys, 

4. Personnel or client assessment, 

5. Systematic expert judgment, 

6. Clinical or case studies, 

7. Informal observation and testimony. 

The real richness of ideas presented by Anderson and 

Ball involve their emphasis on feedback which they refer to 
as the: Communication and Dissemination of Results. They 
emphasive the need for communications of the results to be 
bi-directional. The effective communications of results are 
viewed as a sign of the positive health of the program. Evalu- 
ators should take pride in presenting and disseminating their 
results. The dissemination is more than merely a phase tacked 
on the end of a program; it involves more than simply telling 
the findings. At aminimum, what the dissemination involves 
is that the results should include information about the evalu- 
ation plans and procedures as well as the findings. There are 
also several a een provided to guide the dissemination to 
various audiences. First, if an evaluation is worth doing, 
other groups than the evaluated organization have some inter- 
est in finding out about the results. Second, is that, given 
different audiences, several mediums/methods of dissemination 
may be called for. Anderson and Ball classify some fifteen 
different audiences that should at least be considered for 


dissemination of results [Ref. 33]. 
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Use of results is another consideration once the evalu- 
ation is completed. The logical expectation is that the 
decision maker will use the results to make rational future 
decisions. Weiss lists several reasons why that is not always 
the case. The nature of the organization is sometimes opposed 
to the use of the results; however, if the decision maker and 
the evaluator have maintained close communications throughout 
the process, the potential objections can be reduced [Refs. 33, 
34]. Particular consideration must be given not only to factual 
evaluation of each program, but also the salesmanship aspect 
of presenting the evaluations. 

Anderson and Ball have adapted, from a paper by J. S. 
Berke a structure which they suggest should be applied when 
writing evaluation reports. 

-Brevity and clarity. Critical findings should be 
summarized clearly and simply at the outset. 
-Timeliness--to be useful and utilized, results must 

be reported according to other peoples' schedules and 

not the evaluator's research clock. 
‘Interim products and reports--these help prepare 

decision makers for the impact of larger, later evalua- 
tion reports. Besides, they can allow preliminary 
planning for utilization even before the final report 

is available. 
-Responsiveness--he notes here that traditional researchers 
tend to make questions more interesting, design more 
elegant, analysis more comprehensive, and utilization 

of recommendations more guarded than necessary. 

Pref. 33: pp. 107-03) 

In summary the program evaluator has the responsibility 


to push the results of the evaluation and to bring them to 


the attention of all those who should be made aware [Ref. 33]. 
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ieeermitacal ineradent Evaluation 

Critical Incident Evaluation Program involves manage- 
ment writing regular reports on the performance they observe. 
The critical incident process has as its roots the idea that 
actual behavior should be appraised, not traits of behavior. 
It is an attempt to justify ratings based on specific incidents 
that provide support evidence for the evaluation. Levinson 
places strong emphasis on the need to evaluate on how things 
get done and not just results [Ref. 36]. Levinson argues that 
in reality people are evaluated on the "how", but many are 
led to believe that they are just judged based on results. 

Thus, the profit oriented manager pulls his company 
eee or the "red", but is criticized for the methods used to 
do it. Levinson cites many deficiencies with current evalua- 
tion systems. One is that no matter how watieds Fined the dimen- 
Sions for appraising performance on a quantitative basis, 
judgments on performance are always subjective and impression- 
istic [Ref. 36]. Second, delay in giving feedback creates 
frustration both when performance is good and credit is de- 
served, and when performance is bad and criticism is rendered 
long after the performance. Levinson views performance evalua- 
tion not as a technique, but a process involving both people 
and data. Thus, the process is inadequate [Ref. 36]. 

Levinson proposes that an effective critical incident 
model be composed of the following [Ref. 36]: 


1. A dynamic job description--one which amplifies 
statements of job responsibility and desired 
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outcome by describing the emotional and behavioral 
topography of the task. 


2. A critical incident process--this requires jointly 
setting objectives and discussing each piece of 
behavior that is judged good or bad by the manager. 

3. A psychological support system--to accomplish 
this, the manager must learn to cope with feelings 
Or guilt over the appraisal. He calls this upward 
appraisal concept in which managers who develop 
employees through their effective appraisal should 
be compensated [Ref. 36] 

An interesting aspect was observed by Winton-Oberg 
while observing the use of the critical incident process at 
General Electric: 

People who received honest but negative feedback are 

typically not motivated to do better and often do 

worse after appraisal interviews. [Ref. 35: p. 64] 

The whole critical incident process involves a con- 
tinuous flow of feedback to the evaluated employee. 

Although the critical incident process by its very 
nature should lead to a continuous review and communication 
on positive and negative behavior, that is not always the 
ease. Also the critical incident itself is often a subjec- 
tive evaluation on the part of the supervisor. The system 
should focus attention on actual behavior rather than on 
employee traits. Also it provides an opportunity for the 
employee to find out specifically how to perform if he wants 
to be rated higher the next time. This system does encourage 
regular feedback since the supervisor can hardly wait to the 


end of a period to feedback what he sees as a critical inci- 


dent [Ref. 35]. One of the major factors supporting this 
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system is that the critical incident process takes the sur- 
prise out of the annual or semi-annual evaluation. All those 
incidents viewed by the supervisor as critical would have 
already been brought to the attention of and discussed with 
the subordinate. This technique provides more regular feed- 
back to the subordinate because both good and bad incidents 
should be noted continually. Also if the employee feels un- 
fairly judged, he may appeal the criticism immediately rather 


than waiting a long period of time [Ref. 36]. 


foe SCIric EVALUATION MODELS APPLIED TO THE MILITARY 

The approaches to the evaluation process discussed thus 
maumeieave Deen general in nature. However, in this study evalu- 
ation processes specific to military environments were also 
reviewed. Following are four evaluation approaches to the 
military which were reviewed in detail. 

1. Army Training and Evaluation Program (ARTEP) 

The Army Training and Evaluation Program (ARTEP) for 
Infantry Battalions was reviewed during this research. In 
addition a detailed brief was provided by the G-3, 7th Infantry 
Division [Ref. 37]. The Army Program specifics are laid out 
for Infantry, Airborne, Air Assault, and Ranger units in one 
volume [Ref. 38]. That guide is further supported by local 
guidance at each command. 

The Infantry ARTEP Evaluation has four purposes: (other 
references exist for ARTEP application to Armor, Mechanized 


Harts, etc.): 
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4, 


Establish infantry unit training missions with 
Specified tasks, conditions, and standards of 
performance for combat-critical missions. 

Under simulated combat conditions, train and 
evaluate the ability of the unit. 

Evaluate the effectiveness of past training of all 
echelons of the Battalion. 


Assess future training needs. [Ref. 38] 


The evaluation is broken into three phases: 


ie 
2. 


B'. 


Critigue of Leaders 
Evaluation of Whole Battalion 


Written Report 


The ARTEP employs the following philosophy: 


a 


The evaluation is conducted two levels down (Division 
evaluates Battalion) 

Battalion ARTEP's are scheduled every 18 months. 
Companies/Platoons/Squadrons are evaluated annually. 
Subunit evaluators focus on the end evaluation. 
Evaluations are scheduled in sequential progression 
(squad, platoon, company, then battalion) 

Scheduled preparation time is five weeks for a Battalion 
Fence off the unit for two weeks following the exer- 
cise (no other requirements are placed on the unit 
during this period) 

Use operating force tactics 

Do not evaluate during other training or field 


exercises. [Ref. 37] 
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The ARTEP Evaluations are strictly conducted within the 
divisional unit. The Assistant Division Commander is desig- 
nated "exercise director," but a sister battalion is desig- 
nated as the force to provide the senior evaluator, another 
lieutenant colonel as well as company and other evaluators. 

The ARTEP is viewed as a training evaluation. 

The ARTEP System provides a combination of both objec- 
tive and subjective feedback to the evaluated unit. All of 
the tasks evaluated in the ARTEP evaluation are reported ona 
Satisfactory/Unsatisfactory basis. All unsatisfactory or not 
evaluated ratings must be explained in the remarks. This 
data is placed on a standardized data collection sheet. A 
copy is provided to the evaluated unit shortly after the 
exercise is completed. Additionally a copy of the data sheet 
is forwarded (without any identifying unit data) to the U.S. 
Army Infantry School, Fort Benning, GA. The data forwarded 
to higher headquarters is for the purpose of future training 
development efforts and for use in improving the ARTEP doc- 
trine, devices, and techniques Army-wide [Ref. 38]. The over- 
all evaluation given to a unit is a Satisfactory/Unsatisfactory. 
No percentile scores are associated with or computed for the 
ARTEP results. Thus no comparison is made from unit to unit 
using ARTEP data [Ref. 37]. 

In addition to the written report, a continuous review 
and exchange of comments between the evaluator and unit is 


encouraged. Shortly after the ARTEP is completed, an oral 
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debrief is held for all key personnel of the battalion and 
the evaluators. In some cases the detailed (good) evaluations 
of units to the squad level are announced to all of the troops 
at formation. This technique is viewed as a positive motivation 
for the small units and their leaders [Ref. 37]. 

2 Hunk Comceptual Model of Unit Performance 

A recent study by Steven Funk [Ref. 11] attempted to 
model combat unit effectiveness for the U.S. Army. He found 
many of the same difficulties in defining effectiveness that 
this author found in defining readiness. He described effec- 
tiveness as follows: 

Individual evaluators determine unit effectiveness 

based upon some explicit or intuitive constraint of what 
units are supposed to do and how that is achieved. 

Unit effectiveness is determined by evaluating both 
outcomes and processes, and is determined for perfor- 
mance on all tasks presented to the unit whether they 
are combat related or not. [Ref. ll: p. 19] 

He interviewed groups of officers and senior enlisted, 
primarily battalion commanders, company commanders, staff 
officers, unit officers, and first sergeants. He also reviewed 
a wide array of existing predictive models from the Army, Navy, 
and Air Force. He identified early readiness studies such 
as Army Training Study (ARTS) 1977-1978, which concluded that 
readiness should be viewed from a perspective of relationships 
meng personnel, weapons, equipment, resources, detractors, 


and incentives. He also looked at the Navy's 1974 report on 


readiness which stated, "Organizational effectiveness implies 
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an evaluative or judgmental process against an expectation or 
Seandard” |Ref. ll: p. 1lj. 

The result of his studies concluded that military unit 
effectiveness is far more readily conceptualized than measured 
[Ref. ll]. Part of the problem of measuring military unit 
effectiveness is operationally defining readiness in a peace- 
time environment. 

Funk referenced Etzioni who noted that organizations 
attempt to achieve a balanced distribution of resources across 
needs and did not attempt to maximize satisfaction in one 
area [Ref. 11]. Funk concluded that military units in peace- 
time have great difficulty in maximizing effectiveness or 
readiness because they really face not one objective, Dut a 
whole array of competing priorities all calling for their 
resources. His studies resulted in the proposed conceptual 
model in Figure 3.4. 

This model is referred to as the Unit Performance Sys- 
tems model. It describes behavior as the interrelationships 
of technology, formal and informal unit structure, perceived 
unit requirements and priorities, available resources, unit 
climate, unit process, cause of evaluation of unit actions, 
and the unit operating environment [Ref. 11]. 

EPeicvessect al, Statistical Model of Cembat Effectiveness 

In 1977, a cross-sectional study for the Defense Ad- 
vanced Research Projects Agency was conducted. An effort was 


made to identify through detailed statistical methods key 
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factors which contributed to levels of success in 22 combat 
engagements ranging from World War II to Vietnam. The study 
was aimed at determining the effectiveness of infantry battalions 
based on the judgment of experienced Marine Corps officers in- 
volved in these engagements. Multi-variate statistical analy- 
Sis was used to examine the data results. The study revealed 
that adaptive behavior by units, that is reaction to the com- 
bat environment, was the single most important discriminator 
between successful and unsuccessful performance. They also 
identified three types of activity which appeared to be closely 
related to mission accomplishment: command and planning 
(strongly related), supporting fires (moderately associated), 
and coordination function (relatively weak and appeared to re- 
guire effective command and planning before it made a differ- 
ence) [Ref. 17]. 
This study defined "combat effectiveness" as 

Mmeetne ability of a unlt to accomplish a military 

mission. As such, combat effectiveness refers to 

performance in a hostile environment. [Ref. 17: pp. 1-2] 
As such the focus on the effectiveness of a military unit is 
placed on outcomes and effectiveness and can only be measured 
by mission accomplishment. The infantry battalion was seen 
as not operating in a vacuum, but rather in a world of obsta- 
cles and constraints. Thus the adaptability was an important 
element in performance. Besides adaptability, Hayes identified 
a number of key elements which included: 

‘maneuver during the action 


‘preparatory air, artillery, and naval gunfire 
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‘communications 


‘quality of planning and information as well as others. 
meet. 17: pp. 1-18) 


This study was done during the period when MCCRES was 
being developed. One important observation mace here with 
regard to feedback was that, although MCCRES is comprehensive 
and standardized, a great deal of information is currently 
passing through the hands of the evaluators that could be ex- 
tremely valuable to the evaluated units [Ref. 17]. 

4. Sassone Economic Model of Training Effectiveness 

A fourth approach to effectiveness is taken by Sassone, 
who has taken an economic view which is different from the 
traditional economic approach to evaluating military training 
programs. The traditional approach is to evaluate the train- 
ing on its projected costs and benefits. The Sassone approach 
requires that any new training program be compared directly 
with existing programs that it will replace or amplify. Tra- 
ditionally equipment and training are judged simply on their 
ability to function for the intended purpose. The training 
programs are particularly difficult to evaluate because of 
necessity of measuring the impact of the training on the 
trainee. In the private sector training programs are gener- 
ally judged on their effectiveness by the change in lifetime 
earnings of the trainee after the program [Ref. 39]. If the 
cost of the training program is less than the lifetime increase 
in wages, then this program is judged to be economically 


worthwhile. However, training programs in the military cannot 






be judged this same way, because there is no way to measure 
their contribution toward readiness, sustainability, or effec- 
tiveness since these are not commodities or traded in the 
market place [Ref. 39]. 

Sassone's methodology requires that when a new type of 
training for an individual or unit is developed, it must be 
stated in specific terms to what extent it substitutes for 
existing programs. The constraint in this case is the train- 
ing budget for the specific type of training. There are three 
steps in this methodology [Ref. 39]. 

(1) An equation relating training inputs and outputs must 
be developed. This requires a comparison of the two types 


of training under consideration. (See Figure 3.5.) 





ia | 


A Effectiveness 





Source: Adapted from [Ref. 39: p. 40] 


Pmigicens,5. A Hypothetical Training Input vs. Output 
Relationship 
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The figure shows different levels of effectiveness which can 
be achieved with various combinations of the two types of 
training. The curve shows all the levels of effectiveness 
which can be achieved with the various combinations. The 
shape of the curves describe the substitute ability of one 
m@eaining type for the other. 

(2) Relationship of cost and use data must be obtained. 
There are three types of costs involved: front end costs for 
the training program under evaluation; operational costs, and 
training budgets for the training manager of the program. 

(3) The last step requires the development of a relation- 
ship between training and effectiveness. 

Where potentially greater effectiveness is available 

at greater cost, the issue is simply whether the 

greater effectiveness is worth the greater cost. 

met. 39: p. 41) | 
Sassone fails to specify how’this relation of training and 
effectiveness is obtained. It is assumed by this researcher 
that it refers to some evaluation method such as MCCRES. 
Measures such as those generalized by ARTEP allow for no rela- 
tive measure of effectiveness which is required for this 
type of comparison. 

The cost effectiveness equation is then developed by 
Optimum use of available training resources at each level of 
the training budget. The term "optimum" in Sassone's proposal 
is one that seems to be in conflict with other theorists. 


Funk [Ref. 11] for instance indicates that many competing 
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priorities take resources from the unit thus precluding 
Optimality of resource application. 
Figure 3.6 represents a relation between cost and a 


sample program. 





& Level of effectiveness 


Source: Adapted from [Ref. 39: p. 41] 


Figure 3.6. The Military Value of a Training Program 
Figure 3.6 is described by Sassone as starting at a 
minimal level, indicating that some level of effectiveness 
would exist even without training. The curve initially rises 
rapidly but starts to level off as maximum effectiveness is 
reached. As better training programs are conceived, the mid- 
dle of this curve would shift up. If some new program were a 
perfect substitute of an existing one, the new curve would 


overlay the existing one and be exactly the same curve. Various 
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shifts in the curve up can be caused by a greater level of 
effectiveness achieved for a lower cost. The Figure 3.6 shows 
how the military value of various training programs can be 
compared with other or existing programs. In addition one 
must consider opportunity costs, funds used for research and 
development of new programs which the service could otherwise 
have devoted to existing programs [Ref. 39]. 

The military value of the new training program is repre- 

sented by the increment in the current training budget 

needed to increase effectiveness by the same amount 

as the increase in effectiveness associated with the 

new program. [Ref. 39: p. 42] 
mae Opportunity cost of a new military training program is 
determined by spreading the front end costs equally over the 
units trained per year and the expected life time of the 
meegram applying the appropriate discount rate. Figure 3.6 
is used to compare the ‘military value of a training program by 
using the cost-effectiveness ratio: if the military value 
is less than the opportunity cost, then the new program would 
not be considered economically feasible [Ref. 39]. 

>. Summary 
This section reviewed four approaches to evaluation 

that have recently been developed for applications to the mili- 
tary. The ARTEP employed [Ref. 38] by the Army provided a 
process of evaluation which has much in common with MCCRES 
especially in its application to similar organizations. The 


conceptual model by Funk [Ref. 11] provided a view not only 


of the elements involved in training, but also of some of the 
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barriers and constraints that units must face in a peacetime 
environment. The results of the Hayes et al., Statistical 

Model [Ref. 17] were valuable in that specific characteristics 
were identified that led to success in actual combat engage- 
ments. Finally the Sassone, Economic Training Model was pre- 
sented [Ref. 39] which proposed some new thoughts on the sub- 
ject of evaluating military training programs. The ideas brought 
forth in these studies will be of benefit in the future examina- 


fjmron Of MCCRES. 


FF. FEEDBACK 
...studies have shown that accurate feedback about 
quality of work is a strong stimulus to good perfor- 
mance...People work better when they know how well 


they are doing in relation to some meaningful standard. 
[Ref. 43: p. 174] 


This final section of the chapter will discuss feedback. 
Feedback is variously described by different writers as feed- 
back [Refs. 25,27,36,41], feedback control [{Ref. 15], communi- 
cations [Ref. 32] and further as dissemination, communications, 
oma utilization [{Ref. 33]. 

Whatever the feedback is called it is common to all of 
the models discussed. Feedback is a necessary part of all 
evaluation systems. 

Control to be effective, requires timely, accurate, 
and dependable indicators of effectiveness. Feedback 
should provide information that is adequate to suggest 
appropriate action. To that end, reports must point 
to significant developments, as distinguished from 


what is normal, usual, and to be expected. [Re£. 30: 
em. 504] 
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Flamholtz [Ref. 43] has presented a typical model, Figure 
3.7 of an organizational control system which places emphasis 
on the feedback loop. -He describes feedback as a necessary 


managerial function of the control systems which identifies 


problems [Ref. 43]. 
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Source: Adopted from [Ref. 43: p. 56] 


megure 3.7. Model of Organizational Control System 
This model provides a typical example of how the essential 
Peedback loop functions in an organiZational control or eval- 


uation system. The feedback loop provides an interactive 
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process that allows for routine recycling of the results as 
a comparison of the observed performance to the measurement 
system. The essential element to observe here is that feed- 
back is a standard fixture of the evaluation system. All 
systems viewed that provide a diagrammatic model include a 
feedback mechanism. In addition Anderson and Ball and the 


ARTEP provided detailed structures for feedback. 


G. SUMMARY 

This chapter has covered the topics of management, manage- 
ment control, evaluation, and feedback. These topics can be 
seen as moving from the general to the specific. The bound- 
aries between these topics are not clear cut. There is a 
great deal of overlap in the fields of management, organiza- 
tional control, evaluation, and feedback. This is particularly 
true when they are viewed from different disciplines. For 
instance, much of what is classified in education as evaluation, 
memula be classified under control in organizational or mana- 
gerial theory. A similar situation exists with regard to 
terminology for feedback. Theorists in both the same and 
different disciplines often use various terms to describe 
feedback. 

Since MCCRES attempts to measure or quantify readiness, 
the military models were included. Although the last three 
are static measures or prediction of effectiveness as opposed 
fo MCCRES and ARTEP which are dynamic measures, they do 
provide insight into the various views of effectiveness and 


the observations made from several different disciplines. 
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Feedback was discussed with emphasis on relating the many 
different terms used to describe feedback. In addition the 
feedback loop which is common to ail models reviewed has been 
discussed. The theory of feedback is essential to this study 
and the many ideas and structures placed on it will give much 
insight when applied to MCCRES. Certain elements such as: 
brevity and clarity, timeliness, interim products and re- 
ports, and responsiveness [Ref. 33] have almost universal 
application to feedback theory. The MBO, Evaluation and 
PeenOrity, Critical Incident, and ARTEP all provide a wealth 
of insight on how feedback techniques should be applied. 


The next chapter will provide a detailed overview of the 


MEeCRES System. 
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Vie ee KTS 


In the last chapter detailed examination was made of 
management control, four general evaluation systems, and four 
specific evaluation systems which apply to the military. This 
should provide the reader with a background from which to view 
MCCRES. In this chapter attention is given in detail to the 
MCCRES itself, its development, explanation of the system's 
mission performance standards, and the evaluation process 
applied by MCCRES as well as the reports and feedback designed 
into the current system. The goal of this chapter is to pro- 
vide the reader with a general understanding of how the MCCRES 
is applied, based on the most current Marine Corps directives. 
The purpose of the Marine Corps Combat Readiness Evalua- 
tion System (MCCRES) is to provide a timely and accurate 
evaluation of readiness of Fleet Marine Forces, includ- 
ing reserve units, to accomplish assigned missions. 
[Ref. l: p. 1-A-1) 

The formal MCCRES must be given at least once every two years 


for all FMF units for which performance standards have been 


written. 


A. DEVELOPMENT AND STRUCTURE 

The development of the MCCRES was begun with a review of 
the various combat readiness measurements then currently in 
use. Many of these consisted of individual, as well as unit, 
evaluations and were in the form of inspections which stressed 


appearance more than combat proficiency [Ref. 1]. 
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The MCCRES was structured to access the ability of all 
Marine Corps Combat units: air and ground combat, combat 
support, and combat service support. Specifically it was 
designed to provide: 

-Performance standards based on assigned missions 
-A standardized evaluation process 
-A standardized reporting system 


-Feedback to units indicating strengths and weaknesses. 
[Ref. l: p. 1-A-2] 


The MCCRES was designed to provide a detailed analysis of 
the units' operational capabilities and to specifically iden- 
tify strengths and weaknesses through comparison with doctrine 
under simulated combat conditions. The doctrine is embodied 
in a checklist of requirements. The hierarchy of these quali- 
ties is identified from general to specific as Mission Per- 
formance Standards (MPS), each of which consists of three 
parts: Tasks, Conditions, and Requirements. The MPS set the 
Commandant's acceptable standards for tactical performance 
throughout the Marine Corps. Based on comparison of the 
unit's performance to the MPS under simulated conditions, a 
determination is made of "Combat Ready/Not Combat Ready." 

The MCCRES is designed for both formal and informal evaluation. 
The informal evaluation can be provided by the unit itself 
measuring its performance standards. The formal evaluation is 
accomplished by higher-level commands normally evaluating a 
selection of MPS. For the formal evaluation certain MPS 


are required in all evaluations and others are selected by the 
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organization directing the evaluation based on detailed 

Guidance in Ref. 1. A cross section of MPS‘s are used to 

provide a statement of the organization's operational readiness. 
The evaluators employed in the formal system must be ex- 

perienced, capable personnel. The overall system is designed 

to identify specific operational deficiencies and the nature 

of any latent or potential problems. The MCCRES cycle is des- 


cribed in Figure 4-l. 


Mission 


PERFORMANCE 
STANDARDS 


Unie Irarning 
Objectives 


ra — 


Personnel Logistics 


os a 







Agel Ug lsliexe) 
Psoograms 
EVALUATION ree 
REPORT 
ss vstem 
Feedback 


Bemree: Adopted from [Ref. l: p. I-A-6] 
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B. MISSION PERFORMANCE STANDARDS (MPS) 

In building the MCCRES framework it was decided that the 
evaluation would be structured around a set of Mission Per- 
formance Standards that establish, on a Marine Corps-wide 
basis, acceptable Serer performance. Derivation of these 
standards was accomplished by using five interrelated concepts 
concerning determination of combat readiness. The concepts 
are: 

- Standards must be objective 
- Standards must define for the evaluator what quality means 


- Standards must be based on published doctrine 


- Standards must involve the performance of individual 
Marines in evaluation of unit combat readiness 


~-~ Standards must be simple for the evaluator. [Ref. 1] 

The MPS are defined in detail in Volumes II through VIII 
of Marine Corps Order 3501.2. Each of these volumes pertains 
tO a Specific type of unit to be evaluated, for instance 
Volume II pertains to Infantry and Volume III to Rotary Wing 
Observation Squadrons. In this way the volume lists the MPS's 
involved for each unique type of unit. 

Each MPS is further broken down into three major parts: 

-"The Task to be performed 
-The Condition under which the task is to be performed 


-The Requirements which must be accomplished to successfully 
fulfill the task." (Ref. 1: p. I-B4] 


Examples of the MPS, Tasks, Conditions, and Requirements 


are provided in Figure 4.2. 
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To assist the Evaluator when the Requirement does not 
completely define the quality of performance necessary, Key 
Indicators (KI) are provided. These KI's are defined to clarify 
and explain the requirements [Ref. 1]. The Key Indicator (KI) 
is further defined in Figure 4.3, Key Indicators for Evaluator. 

For simplicity and to reduce the need for subjective judg- 
ments, all MPS's were designed to permit the Evaluator only 
three possible outcomes for each requirement: Yes/No/Not 
Applicable. Those MPS’s dealing with the performance of indi- 
vidual Marines, Exercise of Command and Control, and Fire 
Support Coordination must be used during every evaluation. The 
reason that all MPS's are not employed during each evaluation 
is a limitation of time and resources [Ref. 1]. 

In addition to the evaluation of MPS's which use a simu- 
lated combat environment against an aggressor force, a series 
of MCCRES Standard Performance Tests (SPT) have also been 
developed. Examples of these types of proficiency tests are: 
Foot Mobility Test, Dragon Gunner's Test, Engineer Route Recon- 
naissance Test, and Aircraft Recognition Test. The SPT's are 
used to support the MPS's and are tested objectively prior 
to the MCCRES operational/readiness test. The SPT is formated 
exactly as the MPS; however, a 'Yes' must be achieved in each 
area before the MCCRES is continued. Any unit not found 
Peeererent in the SPT portion of MCCRES will not be tested on 


mae tactical portion. 
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KEY INDICATORS FOR EVALUATOR 


WARNING ORDER 
MUST INCLUDE: 


General information on the situation. 
Units to make the move and anticipated sequence. 
Anticipated time of move. 


Anticipated route and destination. 


STAFF COORDINATION 
MUST SPECIFICALLY ADDRESS: 


Route reconnaissance--map, physical, or aerial photo. 
Ee@pport needed from higher hq--air, vehicle, or firepower. 
Fire support coverage throughout movement. 

Cover and concealment available enroute. 


Logistic aspects affected by location change. 


BASIC PLAN 
MUST BE BASED ON: 


Movement at speed desired by higher hq. 
Control of all elements during move. 
Security of all movement serials. 


Known enemy capabilities. 


MOVEMENT ORDER 
MUST CONTAIN: 


Clearly stated mission. 


Definition of all control measures to be used: check points, 
phase lines, march objectives, etc. 


Specific missions for attached and/or support elements: 
tanks, engineers, TOW, etc. 


Identification of initial point from which move will begin. 


Assignment to march serials. 
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Time of departure. 
Clear identification of available outside support. 
Detailed security procedures. 


As much information on threat as iS available: 
Emphasis on specialized weaponry that can affect the 
move; ATGM, artillery, air, etc. 


Source: Adopted from [{Ref. 1.P: p. I-B-6] 


Figure 4.3. Sample of Key Indicators for Evaluator 
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C. PERSONNEL ASSIGNMENT FOR THE EVALUATION 

The key to successful application of each MCCRES is the 
evaluation process. Selection and training of evaluators is 
at the very heart of the evaluation process. Credibility of 
the evaluators is essential to the success of MCCRES. The 
evaluators selected are to have the requisite skills and recent 
experience as to preclude the need for a long, detailed school 
for the evaluators. However, as needed and as time allows, 
training is provided to evaluators. Since all evluation sys- 
tems require some judgment on the part of the evaluators, and 
MCCRES is no exception, sound judgment is a prerequisite for 
each evaluator chosen. 

The evaluation is structured with the Commanding General's 
Fleet Marine Force Pacific (FMFPAC) and Fleet Marine Force 
Atlantic (FMFLANT) as the Evaluation/Exercise Commander(s) 
mae) for the initiation and conduct of all formal MCCRES 
Evaluations. The responsibilities of the Evaluation/Exercise 
Commanders are outlined in detail in Ref. l. 

1. Evaluation/Exercise Director (ED) 

ED will be designated by the Exercise Commancer (EC) 
and 1s responsible to prepare for, conduct, and report the formal 
MCCRES evaluations. Normally the Commanging General of the 
respective Marine Division or Wing whose subunit is to be 
evaluated is designated ED. 

mee Lacewcalwexercise Controller (TEC) 

TEC will be designated by the ED, along with an ap- 


pointed staff, to serve as the control agency for conducting 
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the exercise. The TEC compiles and analyzes the results of 
the evaluation. The TEC is also responsible for the detailed 
training of the evaluators as well as development of a detailed 
exercise scenario. At the end of the evaluation a formal re- 
port is prepared for the ED describing the combat readiness 
of the evaluated unit. A detailed critique should also be 
conducted for all involved in the evaluation. In the critique 
the results of the evaluated MPS are to be highlighted with 
respective strengths and weaknesses. 
S. Evaluators 

Evaluators should be prepared for the role through 
successful past professional experiences as well as through 
any detailed school provided. The MCCRES evaluators have 
three roles: exercise controller, umpire, and performance 
evaluator. As a performance evaluator, they apply the de- 
tailed MPS's contained in the appropriate volume for the unit 
and they are evaluating and actually make the 'Yes/No/Not 
Applicable' determinations for individual tasks. The evalua- 
tors must possess a complete and thorough understanding of 
the Mission Performance Standards being evaluated. The evalu- 
ators must make any notes needed on an Evaluator Work Sheet 
that are necessary to support judgment during the process 
beet. lj. 

4. Senior Evaluator 

Senior Evaluators will be determined by rank. Evalu- 

ator for the unit evaluated compiles the data sheets from all 


evaluators and should conduct a post exercise wrap up. Any 
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questions or conflicts should be resolved at that time. The 
process must result in the senior evaluator being assured that 
his data provides an accurate reflection of the overall unit 
performance. The senior evaluator then makes the determina- 
tion of the overall unit evaluation of "Combat Ready/Not 
Combat Ready". The senior evaluator provides the data sheets 
to TEC who compiles the detailed analysis and presents it to 
the evaluation/exercise director [Ref. 1]. 

5. Evaluation Staffing 

Evaluation Staffing is the responsibility of the ED 

mer ooth selecting and training all evaluators prior to a 
MCCRES. It is desirable to obtain evaluators from adjacent com- 


mands, not directly related to the organization being evaluated. 


D. COMPUTER ADAPTED MCCRES 

The Marine Corps Combat Readiness System Software Applica- 
fen (MCCRESSA), 1s designed to furnish all organizations in- 
volved with MCCRES a means of assessing and analyzing data 
pertaining to MCCRES evaluations. Specifically it provides 
the following: 


A list of mission performance standards, tasks, and 
requirements. 


...the chief evaluator with a rapid overview and selected 
analysis of the units evaluation. 


Identifies unit readiness based upon mission performance 
standards tested within the applied scenario. 


Provides unit commanders a list of unit readiness deficien- 


cies to initiate corrective action or obtain higher command 
level assistance. 


80 





Reviews unit evaluations to determine validity of mission 
performance standards and doctrine. 


Reviews unit's evaluations to determine limitation/inade- 
quacies in officer education programs. [Ref. l: p. I-E2] 


All of the formal evaluation reports are input te the Head- 


quarters Marine Corps data base [Ref. 1]. 


im REPORTS 
At the end of a MCCRES, after the Evaluator Data Sheets 
have been compiled for all units evaluated, the TEC and senior 
evaluators should review the sheets and make a determination 
of "Combat Ready/Not Combat Ready" for each unit evaluated. 
This recommendation is based on an initial review of "Yes's/ 
No's" for all requirements, taking into account any tasks or 
MPS's that were "demand elements that must be judged Yes for 
an overall ‘Combat Ready' evaluation to be given." 
Ll. Feedback 
"Tne primary purpose of MCCRES reports is to provide 
the feedback necessary for commanders to initiate corrective 
@et1on that will improve combat readiness" [Ref. l: p. 1-Dl]. 
The results should be reflected in improved training 
objectives and may also affect resource allocation of per- 
sonnel, equipment, or logistics support. To assist in the 
reallocation of critical resources, a detailed report is pro- 
vided through the chain of command to the unit evaluated. 
This report provides a short subjective comment as well as 


percentile scores and weighting for each Section, MPS, Task, 
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and Requirement evaluated. (See Appendix A--Sample MCCRESSA 
Perinctcout for one MPS.) 

Additionally a report is provided in message format 
to the Commandant of the Marine Corps (CMC) within 10 days 
following the end of an evaluation indicating the overall 
evaluation "Combat Ready/Not Combat Ready” and unit identifi- 
cation. Within 30 days a detailed report of the MCCRESSA 
printout 1s provided to the CMC. The report should include 
comments or recommendations for improvement or revisions to 
the MCCRES. The purpose of the follow up report is to allow 
CMC to: provide assistance in the review of doctrine, tactics, 
techniques, education and training programs, and validation 
of MCCRES elements. This report also serves to highlight 
trends an repeated deficiencies, and permit analysis for 
corrective action to take place [Ref. 1]. 

Ea FO lilcy 

Based on Marine Corps policy, MCCRES reports are not 
designed to be used to compare the combat readiness of various 
units. This is a result of differences in unit type, Services, 
environmental conditions, and resource allocations. The 
evaluation indexes on MCCRES reports are provided to indicate 
an approximate status of the unit's combat readiness during 
a particular evaluation [Ref. 1]. It can be said that the 
MCCRES provides a snap shot of the unit at the time of the 


evaluation. 
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i. oUMMARY 


This chapter presented an overview of the MCCRES including 
a discussion of the Development and Structure, Mission Per- 
formance Standards, Personnel Assignment for the Evaluation, 
Computer Adapted MCCRES and Reports. The purpose was to 
provide the reader with a general background of how the 
MCCRES is intended to be implemented throughout the Marine 


Corps, based on systems design and policies. 
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V. METHOD OF ANALYSIS 


The research method used had three distinct phases. First 
a general literature review was conducted. Second a specific 
literature review of MCCRES and related defense publications 
was undertaken. Third a detailed field study was made to 
obtain first hand information from those involved with the 


development, application, and oversite of MCCRES. 


A. AUDIENCE 

Anderson and Ball specified in their findings [Ref. 36] 
of the importance of identifying the audience when researching 
and distributing results. To this end the primary audience 
for this study are the planners who conduct periodic reviews 
and Me esone of MCCRES doctrine, that is, the Readiness Branch 
at HQOMC. Additionally the audience should include all those 
involved with MCCRES or who will be involved with MCCRES, as 
well as commanders, evaluators, planners in organizations and 
others who waeuld benefit from understanding the findings pre- 


sented herein. 


B. GENERAL LITERATURE REVIEW 

The results of the general literature review are presented 
in Chapter III. A background in management control, general 
evaluation systems as well as specific examples applied to 


the military from various disciplines is necessary to gain 
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insights into MCCRES. It is evident from Chapter IV that 
MCCRES has incorporated into it many of the positive aspects 
of general evaluation systems theory [Refs. 5,6,20,25,26, 


p27 33]. 


C. SPECIFIC LITERATURE REVIEW 

The specific literature review consisted of articles on 
MCCRES appearing in the various defense publications [Refs. 2, 
6,10,13,31,37, and 38] as well as the technical reports and 
articles available on MCCRES and other military training and 
readiness evaluation systems [Refs. 4,7,11,12,17,38]. This 
specific literature review not only provided a basis for 
understanding the MCCRES from the standpoint of its proponents 
and detractors, but it also provided some valuable ideas on 
alternatives to MCCRES. The detailed technical reports made 
available by George Washington University were invaluable in 
understanding the background and initial development of the 


MCCRES [Refs. 3,4,10]. 


Bee EF LELD STUDY 

The field study was conducted to obtain first hand infor- 
mation on MCCRES. Personal interviews, telephone interviews, 
and a visit to the Readiness Branch HQMC, George Washington 
University, and to the Army's 7th Infantry Division were em- 
ployed to that end. The interviews were semi-structured. The 
stated purpose of the interviews was not just to answer the 


Specific questions posed, but to stimulate thought and identify 
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as Many ideas as possible on how the MCCRES could be improved. 
Although the scope of this project was limited by its primary 
emphasis on feedback, other valuable information was elicited. 
ima cano le 

The sample group chosen consisted of those who had 
the most experience and most vested interest in the MCCRES 
feedback. Because the MCCRES evaluations are most often 
given to infantry battalions and aviation squadrons, these 
were identified as the primary target group. Battalion and 
Squadron commanders, senior evaluators, and division/squadron 
MCCRES officers were identified as the primary sources of 
information since they have the most detailed involvement with 
the MCCRES feedback. Upon interviewing the sample it was also 
found that many of the officers had been involved in several 
MCCRES's and in different capacities. Thus many had experi- 
ence as both a senior evaluator and a battalion or squadron 
commander of an evaluated unit. Divisional or squadron MCCRES 
officers were also a rich source of information because of 
their daily involvement with MCCRES. The secondary source of 
information were "others involved with MCCRES." This group 
is composed of members of evaluated units other than the 
unit commanders, such as unit operations officer, officers in- 
volved with subunits evaluated, and company commanders. They 
also provided much information. 

Given the sample identified, the field study, there- 


fore, concentrated on those of rank from captain to colonel 
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tamene mulitary c©ccupational fieldSof infantry, aviation 
(fixed wing and helicopters), tanks, artillery, engineer, 
and air control. A total of 37 individuals were interviewed. 
The sample group is presented by rank and occupational field 


below in Table 5.1. 


TASGE See 


Rank/Occupational Field Sample Distribution 


al 

mn A o 

S hy 

f fy & $y 
OCCUP A 3 2 =I & S Kd 
TELD | . - f aN 4 5 

RANK 4 fa ~ « 5 x 
Colonel 3 1 0 0 0 0 0 4 
Lt. Colonel 10 2 3 0 in il l 19 
Major 3 4 2 il 0 0 0 en 
Captain D 0 0 1 0 7 0 3 
TOTAL: 18 g 6 2 l 1 1 a7 


These officers are currently serving in billets throughout 
the Marine Corps, representing all divisions and wings, as 
well as the lst Marine Brigade in Hawaii. Other active duty 
officers serving in billets at Headquarters Marine Corps, 
Naval Postgraduate School, and with the reserve establishment 
were also interviewed. 
2. Interview Techniques 
All those interviewed were asked a standard set of 


open-ended questions [Ref. 45]. These questions were 
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purposefully left broad and open-ended to solicit as many 
ideas and opinions as possible. Each interview was conducted 
in person or by telephone by this researcher. All interview 
results were recorded on an answer form and interviewers were 
taped after the permission of the respondent was given. When 
a respondent was uncertain or uncomfortable in anSwering a 
question, he was asked to skip over it. However, no attempt 
was made to limit the respondent to any set of prearranged 
answers. In all cases, where answer choices were posed, 
an invitation for "other ideas or suggestions" was 
always made. As the interviews progressed, the questions were 
refined to help the interviewee narrow in on the specific 
area being researched. 
3. Questions Posed 

The following set of questions was posed to all those 
interviewed: 

(1) When the computerized data results from a MCCRES are 
provided to the evaluated unit, what uses are made of 
that data? 

(2) Do you feel that information/data provided to the 
evaluated unit as the result of a MCCRES are satisfac- 
tory for the purpose of initiating corrective action 
that will improve combat readiness? 

(3) Is the current procedure of providing the results as 
a computer printout based on the evaluation of the 


Sections, MPS, Tasks, and Requirements the most 
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effective format for displaying the results to the 
evaluated unit commander? 

(4) Does the format lend itself to a complete understanding 
of how the unit performed compared with MCCRES criteria? 

(5) Do Marine Corps Directives provide sufficient guidance 
EOmminberoretatdon Of results? 

(6) Would it be helpful to know how your unit performed on 
a Section/MPS/Task/Requirement as compared with other 
Dies mE anple. “YOUr Inrantry battalion scored a 
PSteousthewMiPs, ‘Command and Control Operations," 

This compares with a median of 68% for all other infantry 
units and 71% for all units evaluated. 

(7) What do you feel is the most effective forum for pro- 
viding MCCRES results to the unit tested? 

a. Oral debrief at ea of exercise 
b. Written debrief at end of exercise 
c. Progressive oral debrief throughout the exercise 
dad. Combination of a-c above 
e. Some other method than those suggested 
(8) Who should be made aware of the MCCRES results? 
a. Chain of command 
bee Ouner Like=Units (infantry, air, artillery, etc.) 
c. All Marine Corps units 
a. Background Visits 
Additionally visits were made to Headquarters Marine 


Corps (HQMC), Readiness Branch, George Washington University 
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poeeeO Ene Army's 7th Intantry Division, Fort Ord, California. 
Individuals interviewed during these visits provided many 
valuable insights into the development and overseeing of 
MCCRES, as well as a detailed understanding of how the Army 
ARTEP system is employed. Several of the individuals inter- 
viewed were involved in the early development of the MCCRES: 
Professor W. H. Marlowe, George Washington University; Lt. 
Colonel Paul Catalone, USMC, and Colonel M. P. Sullivan, USMC 
(telephone interview), and Lt. Colonel R. S. Gibson, USMC 


(telephone interview). 


E. CONTENT ANALYSIS/DATA REVIEW 

since the material/responses to the questions asked were 
subjective in nature, content analysis was used to analyze 
the results [Ref. 46]. 

i COmeencemnalysis Defined 

Berelson states: "Systematic content analysis attempts 
to define casual descriptions of the content, so as to show 
objectively the nature and relative strength of the stimuli 
applied to the reader or listener" (Ref. 46: p. 14]. 

This procedure is employed by constructing an analysis 
outline which described by Cartwright embodies the following 
general principles: 

Step l--specify data needs, 
Step 2--map out plans for tabulation, 
Step 3--map a skeleton of the outline, 


Step 4--fill in categories for each variable, 
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Step 5--establish procedures for using the material, 


Step 6--try out the analysis outline and use procedure. 
[Ref. 45: pp. 454-460] 


The application of content analysis allows the researcher 
to study the communications and focus on the interaction through 
messages which link communicating parties. Not all signifi- 
cance can be extracted by inspection or mere observation. 

The real purpose of any analysis is to illuminate and make 
inferences about something that is not otherwise apparent. 
Thus the process of content analysis is a process like the 
congruence definition of evaluation which employs objectives 
and criteria neato pene a measurement process which is 
both scientific and objective. The process is not, however, 
totally devoid of judgment and does allow the researcher dis- 
cretion in categorizing results [Ref. 47]. 

Beenanalysis OF Specific Responses to Questions 

The purpose of content analysis 1s to provide some 
means of quantifying subjective information. To accomplish 
this rules must be established. The data contained in the 
response to each of the eight questions was treated as a 
separate unit and no attempt was made during the interview to 
relate any answers from one question to that of another. This 
procedure was used to solicit as much information possible and 
so as not to limit any of the respondents in their comments. 
Since each question stood alone and was relatively subjective 
in nature, each question provided an opportunity for a response 


to contain similar information in response to different 
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questions. Thus once a respondent began to provide information 
to a question which was more appropriately categorized as a 
response to a different question, the interviewer SS re- 
corded the information under the question currently being 
asked. However as part of the analysis of the interviews, 
responses to one question that were relevant to other questions 
were considered. 

Each question had certain categories set up for the 
responses and each response made was placed in one of these 
categories. In order to generate useful information from con- 
tent analysis, the categories must be collectively exhaustive 
ama mutually exclusive. For example: 

Question #1l--When the computerized data results from a MCCRES 
are provided to the evaluated unit, what uses are made of 
that data? 
Responses to this question fell into three groupings: 
"none" 
"used for planning future training and resource allocation" 
"historical reference only" 

It should be understood that each MCCRES has many key 
participants and each has a different involvement in the 
evaluation. Some such as the unit commander and senior evalu- 
ator are involved virtually in all aspects. Others such as 
the operations officers and operations evaluators have a more 
limited focus in their responsibilities. Thus a system was 
developed to give varying weight to those who had differing 


levels of involvement with MCCRES. 
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A scoring system was established with the following 


weighting: 
Intensity Points 
High 3 
Medium 2 
Low al 


Bon Intensity—--those involved directly in one or more 
MCCRES as unit commander, division or 
wing MCCRES officer, senior evaluator. 

Peawumeintensaty——those involved with less direct 
responsibility for results: operations 
officers, other evaluators, company 
commanders, etc. 

Bevel tnvensity—--Employed in only one case where interviewee 


was involved ina forerunner evaluation to 
MCCRES 


eeonatveis Or General Responses to Questions 
To determine which guestion responses should be dis- 
played for the reader, a 10% rule was formulated. If 10% or 
more of the respondents to a question amplified their specific 
answer with similar comments or suggestions, then the comments 
were included. Thus, any of the general responses included 
in the next chapter were mentioned by four or more of those 


sampled. 


F. SUMMARY 

In this chapter the research method is summarized. A 
total of 37 Marine officers were interviewed. Although the 
number is relatively small, it should be remembered that the 


Marine Corps has three infantry divisions and three aircraft 
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wings on active duty. Therefore, the total population of those 
qualified to answer the questions is drawn from a limited 
population of officers in the target group. It is hard to 
estimate the total number of officers that have been involved 
in these primary capacities since the billet holders rotate 
regularly; but it is safe to say that only a few hundred offi- 
cers have served as battalion/squadron commanders or senior 
evaluators of evaluated units. Since the MCCRES is only re- 
quired to be given every two years, the sample used appears 


to be reasonable. 
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Vp oo lS 


This chapter presents the detailed analysis of the results 
obtained through the field study. The specific and general 
responses to questions posed were content analyzed. The basis 
for this type of analysis is provided by Berelson [Ref. 46]. 

Percentages derived from the content analysis are the 
weighted averages of the points for each category as compared 
with total points for each question. The responses are weighted 
based upon the level of involvement of the respondent in previ- 
Gus MCCRES's. In cases where some interviewees did not answer 
a certain question, the percentage is based only on the re- 
Sures Of those who did answer. NONCerCnee was made to cate- 


gorize reasons why those that failed to respond did so. 


PeeeeoNALYSIS OF SPECIFIC RESPONSES TO QUESTIONS 

First Question: (1) When the computerized data results 
from a MCCRES are provided to the evaluated unit, what uses 
are made of that data? 

Responses to this question were grouped into three categories-- 
None, Used for planning, and Historical reference only--and are 
shown in Table 6.1. The majority of the respondents (63.6%) 
said that the results of MCCRES were used for planning future 
training or specifically for future resource allocation. 

Second Questions: (2) Do you feel that information/data 


provided to the evaluated unit as a result of MCCRES is 
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Bacastactory for the purpose of initiating corrective action 
that will improve combat readiness? 

The responses to this question are categorized as positive 
or negative and are shown in Table 6.2. The majority of the 
respondents (81.3%) said that the information/data provided 
mee a MCCRES is satisfactory for initiating corrective action 
that will improve combat readiness. 

Third Question: (3) Is the current procedure of provid- 
ing the results as a computer printout based on the results 
of Sections, MPS, Tasks, and Requirements, the most effective 
format for displaying the results to the evaluated unit? 

The responses to this question were categorized as posi- 
tive or negative. As shown by Table 6.3, 64.6% of the 
respondents said that the current procedure of providing the 
results as a computer printout was not the most effective 
format for displaying the results to the evaluated unit. 

Fourth Question: (4) Does the format lend itself toa 
complete understanding of how the unit performed compared with 
MCCRES criteria? 

The response to this question was categorized as positive 
or negative. As shown in Table 6.4, 55.6% of the respondents 
said that the current format lends itself to a complete under- 
standing of how the unit compared with MCCRES criteria. 

Fifth Question: (5) Do Marine Corps Directives provide 
sufficient guidance for interpretation of results? 

The responses to this question were categorized as posi- 


tive or negative. As shown by Table 6.5, 72.9% of the 
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respondents said that Marine Corps Directives provide suffi- 
cient guidance for interpretation of results. 

Sixth Question: (6) Would it be helpful to know how your 
unit performed on a Section/MPS/Task/Requirement as compared 
with other units? Example: Your infantry battalion scored 
a 73% on the MPS, "Command and Control Operations." This 
compares with a median of 68% for all other infantry units 
and 71% for all units tested. 

The responses to this question were categorized as posi- 
tive or negative. As shown by Table 6.6, 55.4% of the respondents 
said that it would be helpful to know how their unit performed 
On a Section/MPS/Task/Requirement as compared with other units. 

Seventh Question: (7) What do you feel is the most 
effective forum for providing MCCRES results to the unit? 

This question was evaluated based on selection of one of 
five responses: oral debrief at end of exercise, written at 
end, progressive; combination of above; or other. As indicated 
in Table 6.7, all thirty-seven interviewees responded to the 
question. Of the total, 91.5% responded that a combination 
of oral and written debriefs is the most effective forum for 
providing MCCRES results to the unit evaluated. 

Eighth Question: (8) Who should be made aware of MCCRES 
results? 

The question was evaluated based on selection of one of 
three responses: chain of command, other like units, or all 
units. As indicated in Table 6.8, 60.2% of the respondents 


indicated that all units should be made aware of MCCRES results. 
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The responses to the questions are summarized for the 
reader in Tables 6.9 and 6.10. Table 6.9 shows the distribu- 
tion by rank of the specific responses to the eight questions. 
Table 6.10 shows the distribution by community (ground/air). 
A more detailed display of data by occupational specialty was 
available to the researcher; however, to protect the anony- 
mity of the respondents only the general categorizations of 
answers is displayed. In reviewing the data in Tables 6.9 
and 6.10 there is no direct evidence of bias by any specific 
rank or community. This same result was true when evaluating 


mee data by occupational specialty. 


B. ANALYSIS OF GENERAL RESPONSES TO QUESTIONS 

The primary purpose of this research was to gain insight 
and new ideas. To this end the respondents were specifically 
asked to expand upon and explain their answers. Their explana- 
tions resulted in the following information. The 10% decision 
rule discussed above was used to determine which information 
Zoe present. 

In response to Question One, the majority of the inter- 
viewees said that the results from a MCCRES found their way 
into unit training plans and schedules. One of the most serious 
concerns brought out in response to this question concerned 
timing [Ref. 20] of the evaluation. If the MCCRES is given 
Sixty days or more prior to deployment then the unit may 
experience substantial turnover of key personnel and nullify 


the results of the MCCRES by changing key players. The 


106 





TABiEE 6.9 


Distribution by Rank of Responses to Questions 


Question # Colonel Lt. Colonel Major Captain TOTAL 
1. a. None 0 4 i 0 > 
b. Planning 4 11 4 3 22 
c. Historical 0 5 4 0 fi 
2. a. Positive 8 16 7 3 29 
b. Negative 0 4 2 0 6 
3. a. Positive 1 ii 2 il alas 
b. Negative 2 el: 6 0 19 
4, a. Positive 3 8 3 il 2 
b. Negative 0 9 3 0 12 
5. a. Positive 2 eZ 6 0 20 
b. Negative 0 > fl 1 7 
6. a. Positive 3 10 o 1 Zi 
b. Negative i 9 Th iS 
feeea. Oral. 0 0 
b. Written 0 0 
c. Progressive 0 0 
d. Combination + iy, 10 3 34 
e. Other 0 0 0 0 0 
See a. Chain 1 i el 
b. Like 0 1 2 a 4 
e. All 3 10 7 fl 21 
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Distribution by Community of Responses to Questions 
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Fixed wing, Helicopter, Air Control 






alternative of this presents a serious dilemma. If the MCCRES 
is given too close to time of deployment, then little time 
exists to correct all but the most glaring deficiencies. 
Additionally since no real second evaluation is made of the 
shortcomings, the USMC may be deploying units who have not 
imieract corrected shortcomings identified during MCCRES. 
A second major problem identified by the respondents is 
the timeliness with which the unit gets the actual MCCRES re- 
port. Some organizations present at least a rough of the 
Bernctout to the unit at the time of the formal debrief, usually 
12-24 hours after termination of the MCCRES. Others do not 
peeeive their formal printout until there is a complete staffing 
by division/wing, regiment/group; and in at least two cases 
the unit did not receive their formal feedback until they were 
on their deployment. One unit did not see their printout at 
all after the MCCRES. (This comment is provided as an exception 
to the 10% rule because of the seriousness of this situation.) 
Responses to Question Two indicated that the current 
MCCRES gives sufficient information to improve combat readi- 
ness. In the case of the rotorywing MCCRES, three of the six 
rotorywing respondents indicated that more weight was needed 
on flying skills than is currently given by the MPS in Volume 
III. Generally the responses from the rotor wing community 
indicated too much emphasis is given to non-flight requirements 
aieietcoG little to flight skills. In addition responses from 


the fixed-wing community indicated that technology is changing 
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more rapidly than the MCCRES. With the implementation of the 
FA-18 aircraft and changes to other aircraft, the MCCRES 
evaluation needs to be changed to be current and realistic. 

In other words the aircraft can do a lot more than MCCRES re- 
quires. Discussions with the Readiness Branch indicate they 
are well aware of the problems caused by changing technology 
in the FA-18 and other new aircraft, and are presently improv- 
ing the MCCRES requirements to test the aircraft at limits 
that more closely approximate its actual capabilities rather 
than against the less demanding F-4 Requirements. 

A total of 13 respondents commented that MCCRES results are 
only as credible as the evaluators. The absence of standardi- 
zation of evaluators and evaluator training, as well as the 
variances in how organizations pick evaluators had a great 
impact on the value perceptions and the worth of the MCCRES 
aS a measure of the unit's readiness. This was expressed by 
six of the fifteen officers in the aviation community. Forty 
percent of the respondents submitted an unsolicited comment on 
the quality of the evaluators. If the evaluator is not re- 
garded as highly qualified by his peers, then the evaluation 
is regarded with little weight. The question of evaluator 
bias is discussed in more depth by Wheeler [Ref. 5]. 

A final problem cited dealt with the MCCRES concept of 
comparing unit performance with published combat doctrine. 
Doctrine as codified in written manuals may be outdated. If 


so, the USMC is evaluating units by comparing their performance 
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to standards which would not be employed on the modern battle- 
field. An example of this cited by four infantry commanders 
is "Maneuver warfare". Maneuver warfare 1s widely accepted 
by Western military analysts as the most promising way to 
fight and win any future war against Soviet tactics. However, 
maneuver warfare is not written into current Marine Corps 
doctrinal publications. This shortfall points out the need 
feneaLstributwon of MCCRES results to all training and doctrine 
commands to ensure they are aware of the strengths and short- 
falls that units/individuals display on MCCRES. 

The most common response to Question Three was negative. 
A percentage of 64.6% of the respondents said that they did 
not feel the computerized printout is the best way to dis- 
play MCCRES results. Nineteen of thirty respondents indicated 
more subjective data was needed to fully understand the units' 
evaluations and to be able to take corrective actions. The 
computer printout provides a "Yes/No" evaluation, but it fails 
to provide the "why" and "what" [Ref. 36]. More details are 
needed to describe specific reasons for "No" marks and recom- 
mendations are necessary to help the unit find ways to improve. 
All those responding negatively (19 of the 30 respondents) 
indicated it was much more valuable to spell out the details 
of a shortfall and to suggest corrective action, than to simply 
provide a "Yes/No" on an evaluation sheet. Several innovative 
ideas have come out of this problem. One wing MCCRES officer 


now writes all comments directly on the MCCRES printout before 
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it is provided to the unit. The wing MCCRES officer said 
this facilitates a much more complete understanding of problem 
areas. The general concensus of the 30 respondents to this 
question was that the subjective, explanatory information is 
by far the most valuable to the evaluated unit and the per- 
centile scores on the printout are probably of greater value 
to higher level headquarters for use in planning future training. 
In response to Question Four, 15 of the 27 respondents or 
95.6% agreed that the current format is not difficult to under- 
stand. In general those who indicated a willingness to refer- 
ence the’ directives and compare the printout to the appropri- 
ate Volume were satisfied with the explanations provided 
therein. Those who said the current format is difficult to 
understand (12 of the 27 or 44.4%) did so based on a criterion 
of the inconvenience of the system, that is requiring both 
the printout and the Volume. This requires a good bit of 
tedious research to find the details of each MPS/Task/Require- 
ment. The recent addition of a description of each MPS/Task/ 
Requirement on the MCCRES printout should help reduce this 
problem. On the other ee the brief description may cause 
even less referencing of the source document to obtain complete 
details. One suggestion was that key players from the unit to be 
evaluated should be given detailed classes and instructions on 
the MPS several weeks prior to a scheduled MCCRES. This has 
recently been started in one wing and the results are excellent. 


This contention was supported by both squadron commanders 
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questioned as well as by those in the Readiness Branch at 
Headquarters Marine Corps. Just as there is a need to edu- 
cate evaluators, there is also a need to educate those to be 
evaluated [Refs. 5,25,26]. 

An added point made by five unit commanders was that the 
results must get out to all those concerned. If the response 
to an evaluation is delayed or the results simply become his- 
torical record, those who need to know at company, platoon, 
and squad level are never made aware of the results. Much 
emphasis is needed on informing all participants down to the 
individual Marine of the results of MCCRES [Refs. 15,37]. 
MCCRES results should be distributed to all detachments to 
ensure they know how their performance was evaluated. 

Responses to Question Five indicated that the orders gener- 
ally seem well written and easy to interpret. The same sugges- 
tion for education made on Question Four applies here. That | 
is, any instruction given to key unit players will result in 
better overall understanding of MCCRES and allows the unit 
to better use the results. 

Several specific shortcomings in MCCRES were also noted 
in response to Question Five. Those relating to aviation were 
addressed under Question Two. One infantry division has found 
some serious shortcomings, specifically in the failure to pro- 
vide for intelligence gathering, fire support coordination, 
and allocation of tasks from higher headquarters. These three 
areas all apply to specific MPS's that need expansion cr 


improvement. 
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Question Six by far evoked the most emotional and intense 
responses. Any time a comparative index iS proposed it has 
the potential for both good and bad results depending on its 
use. Of 37 individuals interviewed, 55.4% favored some sort 
of use of comparative scores. Those opposed tended to do so 
based on the view that any comparison of scores provided a 
potential for use as a report card to compare units and 
specifically unit commanders. The primary objections to the 
use of comparative scores was based on the lack of overall 
standardization within MCCRES, especially the differences in 
evaluators, scenarios, terrain, and weather conditions. 

Those who favored comparative scoring GieaesG LOG Var1Ous 
reasons. Four respondents thought comparative scoring had 
fue, but only for higher headquarters, training, and dec- 
trine commands. Seven others thought comparative scoring had 
value if shown to the unit in complete privacy. A final segment 
of ten respondents indicated that the comparative scoring had 
potential to create more unit competition. Thus, overall, only 
10 out of the 37 respondents supported any sort of mass publica- 
tion of comparative scores. 

The results to Question Seven provided evidence that feed- 
back must be provided in various forms throughout the exercise 
to be most effective. As seen in response to Question Three, 
64.6% wanted more subjective feedback. In Question Seven 
that was reaffirmed. The respondents said that by far the most 


valuable feedback was obtained from verbal, subjective critiques 
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given immediately after an event has taken place. 

A total of 91.5% of the respondents indicated a combination 

of debriefs and written results is most desirable. The written 
results should not only include the objective numerical re- 
sults of "Yes/No" markings, but also explanation of all the 
"NO/N.A." markings. 

Question Eight was most often answered with "all units," 
60.2% of the time. The specific point made here was that the 
chain of command has an obvious need to know the results. 

All units could benefit from a generic trend analysis report. 
Trend analysis would provide a cross pollination of critical 
information throughout the Marine Corps and assist those in- 
volved in developing training and doctrine. Those favoring 
Beeeeunles' Or ~Other like units” comprised a total percentage 
of 69.3%, provided a favorable response to the suggestion of 
receiving a generic trend analysis at regular intervals from 
the MCCRES Data Bank. The fact that no respondent provided 
any objection to this idea is indicative of its positive 


value. 


C. SUMMARY 

As can be seen in the next chapter the analysis of specific 
and general responses to the eight questions provided answers 
to the research questions posed by this thesis. In addition 
a number of important unsolicited comments, outside the scope 
of this study, were also obtained. These comments are pre- 


Bemtced in Appendix B. 
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Vitz  ebrSeCUSSC ION AND CONCLUSIONS 


Pee ANSWERS TO PRIMARY RESEARCH QUESTIONS 

The results of the research have provided answers to ques- 
tions posed at the beginning of this project. The three 
primary research questions were: 

(1) After MCCRES, what information should be fed back 

to the evaluated unit commander? 
(2) In what form should the evaluation feedback be? 
(3) What channel(s) should be used to provide the 
feedback to the evaluated unit? 
A comprehensive feedback anoles 1s presented which incorporates 
the results for the three questions. 

It was evident throughout this study that the unit com- 
mander faces many constraints: training time, material, 
equipment, logistics, and personnel. The information from 
MCCRES should be provided in such a way as to allow for 
better allocation of these scarce resources to the unit. 

As discussed in Chapter II, the overall goal of the MCCRES 
momemoc Only to provide a measure of unit readiness, but to 
provide effective feedback to the organization to enhance the 
Opportunities to improve readiness by applying resources to 
areas identified as deficient by MCCRES. The theories under- 


lying management control and evaluation, as presented in 
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Chapter III, provide a rich background for understanding how 
to perform the MCCRES evaluation and how to effectively use 
the results of an evaluation. In both theoretical and applied 
models of evaluation, information and feedback play an impor- 
tant role in both performing and using the results of an 
evaluation. In general the modelS of evaluation have common 
desirable characteristics such as: brevity, clarity, timeli- 
ness, interim products and reports, and responsiveness of 
feedback [Ref. 33]. 

Additionally, characteristics of the organizational control 
system [Ref. 43] such as goals, standards, measurement systen, 
evaluation and reward system as well as the feedback loop play 
an equally important role in the evaluation models [Refs. 30, 
ia oo,3>, and 38]. 

Based upon these findings and the results of a field study, 
a model for MCCRES feedback was developed. Themodel is pre- 
sented in Figure 7.1. Each of the eight major elements of the 


model that affect the evaluated unit are explained below. 


Pre-MCCRES Brief 


Trend Analysis Real Time Feedback 





Follow-up Report Evaluator Worksheets 


Comparative Percentile Two-way Debrief 
Results 


Computerized MCCRES Printout 


Figure 7.1. Model of MCCRES Feedback to an Evaluated Unit 
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1. Pre-MCCRES Briefing 
A detailed orientation several weeks prior to MCCRES 
to provide a firm foundation for all key personnel. Explana- 
tion of appropriate Volumes of MCO 3501.2. Explanation of 
the details of each MPS and procedures for the evaluator 
allows the unit to maximize the potential from the MCCRES 
feedback [Ref. 36]. 
2. Real Time Feedback 
Regular verbal feedback throughout the exercise by 
evaluators at each level. At any logical break in the scenario 
detailed evaluations of the most recent events should be pro- 
vided by the evaluators. This provides data while the event 
is fresh in everyone's minds [Ref. 33]. 
3. Evaluator Worksheets 
The Evaluator Worksheets should be provided directly 
to the evaluated element shortly after conclusion of the 
exercise. This provides an interim evaluation report and 
allows for maximum feedback to the specific element evaluated. 
It would be especially valuable to subunits and attachments 
who often receive only minimal feedback under current proce- 
dures [Ref. 36]. 
4. Two-way Debrief 
At the end of exercise, twelve to twenty-four hours 
after the end of the formal MCCRES, a detailed debrief should 
be held for evaluators and key players from the unit evaluated. 


Opportunities should be given for two-way discussion to resolve 
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any confusion or disagreement particularly regarding subjec- 
tive areas. The individuals from the evaluated unit shouid 
leave with not only a clear understanding of the deficiencies, 
but also with a clear understanding of what caused the "No" 
evaluations and suggestions for improvement [Ref. 33]. Since 
the commander of the unit evaluated is responsible for the 
employment of the feedback, he should have the flexibility 
to tailor the debrief to his leadership style. The commander 
May desire to have only a few selected evaluators and officers 
present or the unit commander may desire all evaluators and 
unit officers to be present. 
Pemecompeuccnnzed MECRES Printout 

The computerized MCCRES printout with percentile scores 
and detailed amplifying remarks should be provided to the 
unit in a timely manner [Ref. 33]. It does a unit little good 
to get the detailed results when there is no time to correct 
deficiencies. What is timely can vary depending on deployment 
schedules, but getting a copy of the results in the hands of 
the evaluated unit so that action can be taken is essential. 
It is inexcusable to allow staffing to delay this critical 
information. Delays also put the unit further away from the 
MCCRES and it tends to become an historical document vice a 
map for future training. 

be Comparative Percentile Results 
The data needed to compare units within a division or 


wing is currently available with limited research within the 
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division or wing files. Additionally most division or wing 
headquarters are well aware of how their scores compare with 
other like units. Usage of general MCCRES data bank to provide 
comparative scores, medians, and standard deviations should be 
Peovided to major headquarters staff, training, and doctrine 
commands for the limited purpose of planning future resource 
allocation and to develop training doctrine. These compara- 
tive results should be limited to Section/MPS/Task scores and 
should not include any sort of comparative final score for 
the units as a whole. Providing any sort of overall scores 
that could be used to compare individual units tested could 
well have a negative impact because of the many variances in 
evaluation conditions that exist. It is further recommended 
that these comparative results be directed only at division/ 
wing level or higher commands to reduce the chance of any 
type of unit comparisons. 

An example of comparative percentile results that could 
be provided to division/wing level for use in planning future 


training is presented in Figure 7.2. 


Eno peel Melt MECRES RESULTS 


Divas ton, Overall 
Wing Marine Corps 
Section Zo Operations Performed FSi se Calais 
MPS DPX Actions by Marines 97.8% 98.43% 
Task Dearie. Discipline 86.4% OO. 25 
Figure 7.2. Sample of Comparative Summary Percentile Results 
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Currently no follow up evaluation is made to con- 
firm that the deficiencies have been corrected. Some type of 
final check is needed. This check would best be accomplished 
with little formality. Since the unit commander has the 
responsibility for application of MCCRES feedback to improve 
unit readiness, it is best left to the unit commander to do 
the final check which certifies that deficiencies have been 
corrected. Resource reallocation from outside the evaluated 
unit likewise should be certified at the appropriate level. 
The key issue here is that the unit must be deployed at the 
highest level of combat readiness and verification of correc- 
Evem otf MCCRES deficiencies at least on an informal basis is 
essential. This follow-up should be made as simply as possi- 
ble. The unit commander can appropriately perform this certi- 
fication based on his own experience and judgment. A formal 
follow-up inspection is neither necessary nor desirable. 

8. Trend Analysis 

To gain a higher level of across-the-board readiness, 
a trend analysis would be of value to all units. The MCCRES 
data base represents a resource that could be used to keep the 
Marine Corps informed of its overall strengths and weaknesses. 
Special emphasis should be made on getting these results 
to training and doctrine organizations throughout the Marine 


Cone 
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B. ANSWERS TO SECONDARY RESEARCH QUESTIONS 

The results of the research provided answers to the second- 
ary research questions posed at the beginning of the project. 
The three secondary research questions were: 

(1) Validity of comparing MCCRES results given under 
differing conditions. 

(2) MCCRES contribution to effective training. 

(3) Time availability and follow up to ensure correction 
of deficiencies identified during MCCRES. 

A first response to the question of the validity of com- 
paring MCCRES results given under differing conditions might 
be to say that such comparisons can only be made if complete 
standardization is achieved. Although complete standardiza- 
tion for purposes of evaluation may seem to be desirable, this 
may not be feasible. Since various Marine Corps units (east 
coast, west coast, overseas) are preparing for different mis- 
sions, it is desirable to simulate their expected combat 
employment through the use of different scenarios, terrain, 
and even MPS's. Additionally the Marine Corps has other exer- 
cises, such as the Combined Arms Exercise or CAC's, which are 
used to evaluate under standard scenario, terrain, and evalua- 
tors, but they have a more limited purpose than MCCRES. Be- 
cause of the broader purposes of MCCRES, complete standardization 
is probably not desirable. In addition it can be questioned 
as to how much standardization is really necessary to Rave an 


effective evaluation system. The performance evaluation system 
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used for promotions is one familiar to all Marine officers. 

It can be argued that the MCCRES certainly has much more 
objectivity than the Performance Evaluation System [Ref. 47]. 
Thus for its intended purpose, MCCRES appears to provide a 
relatively high degree of objectivity when compared with other 
military systems and the results are likely to be useful, at 
least for generic comparisons and to identify trends. 

In response to the question of MCCRES's contribution to 
effective training, it is evident from the data gathered for 
this thesis that MCCRES has the potential to, and currently 
does, improve unit training. Many units train to MCCRES 
standards and go so far as to use actual MCCRES Section/MPS/ 
Task/Requirements to structure their unit training. There- 
fore as the Individual Training Standards (ITS) are developed 
for all units, it is essential that they be closely compared 
with MCCRES standards so that the training and evaluation 
development move in the same direction [Refs. 5,28,29]. 

The third question, time availability to correct MCCRES 
deficiencies, has previously been discussed. No doubt more 
time would allow for more training in deficient areas. How- 
ever, the question of follow up is also critical. At least 
an informal re-validation of deficient areas is needed to 


ensure the effectiveness of the MCCRES. 


C. SUMMARY 
This research has provided detailed answers to the ques- 


tions posed at the beginning of the study. The feedback to 
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the unit is the key element to making the MCCRES of value to 
the unit evaluated. The purpose of MCCRES is to provide timely 
and accurate evaluation of the force readiness. Key to im- 
proving readiness is identification of strengths and weaknesses, 
and providing timely and thorough feedback to the evaluated 
unit and its chain of command. The more timely and thorough 
the feedback, the higher the likelihood of improved future 
readiness. The MCCRES is currently viewed as a sound, valua- 
ble evaluation system. However, as this thesis has demon- 


strated, the feedback to the units can be improved. 


D. STUDY RECOMMENDATIONS 

The feedback model developed and displayed as Figure 7.1 
should be incorporated as the minimum acceptable procedure 
for providing feedback from any MCCRES. To accomplish this 
the eight steps in the feedback model should be followed during 
each evaluation. The following information should be routinely 
provided as feedback. 


Pre-MCCRES Briefing--to key MCCRES players from the evaluated 
Uae 


Real Time Feedback--distributed to all those evaluated 


Two-way Debrief at end of exercise--tailored to needs of 
the unit commander to ensure full understanding of 
evaluation results 


Computerized MCCRES Printout--provided in a timely manner 
to allow new resource allocation prior to deployment 


Comparative Percentile Results--to help direct unit train- 
ing and for incorporation by individual training and 
doctrine commands 


Follow-up Report--done informally by unit commander to 


ensure closure of feedback loop by correction of 
identified deficiencies 


124 






Trend Analysis--to support future training as well as 
changes in doctrine. 


Implementation of these recommendations would suit the 
stated needs of fleet units as well as serve the intended 
MCCRES purpose of providing an evaluation of unit readiness 


to higher headquarters. 


E. RECOMMENDATIONS FOR FUTURE RESEARCH 
The unsolicited comments provided ripe areas for further 
research. Any item listed in Appendix B could be researched 


and could provide information for program improvement. 


F. SUMMARY OF STUDY 

The purpose of this study was to determine the most appro- 
priate and effective manner in which to provide feedback to 
the unit commander from a MCCRES. To this end research was 
conducted into the areas of management control and evaluation 
theory. Additionally specific research was conducted in the 
area of existing systems which measure or evaluate readiness. 

The study was approached from a multi-disciplinary back- 
ground with a detailed emphasis on the economic question of 
how better distribution could be made of the scarce resources 
of manpower, equipment, and training time based on the results 
Of MCCRES. To answer the questions a detailed field study was 
conducted in which interviews were completed with 37 Marine 
officers of rank from Captain to Colonel who have had involve- 


ment with MCCRES as key billet holders. Additionally much of 
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the original documentation that laid the groundwork for the 


development of MCCRES was reviewed along with interviews and 


visits with some of the original developers of the system. 


The results of the study are displayed as an eight step 


feedback model which is based on accepted theory in the fields 


of management control and evaluation theory, as well as the 


results of the field study. These results are communicated 


to the reader as a model in Figure 7.1 which is comprised of 


the following elements: 


(1) 
(2) 
(3) 
(4) 
->) 
(6) 
(7) 
(8) 


Pre-MCCRES Briefing 

Real Time Feedback 

Evaluator worksheets 

Two-way Debrief at the end of Exercise 
Computerized MCCRES Printout 
Comparative Percentile Results 
Follow-up Report 


Trend Analysis 


The incorporation of this model as standard MCCRES 


feedback procedure will significantly enhance the value of 


the results to the evaluated unit and will improve the under- 


standing of the resource allocation needs at all levels. 
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APPENDIX A 


SAMPLE MARINE CORPS COMBAT READINESS EVALUATION 
SYSTEM SOFTWARE PRINTOUT 


2.0 INFANTRY BY SECTION 
SECTION WEIGHT EVAL CUM WT 
A) OPNPERFORM 100 .00 95 68.97 
B) STDTACTICS 45.00 92 31.03 
C) SPECIALOPS 18.00 N/A 0.00 
D) EXTERSUPTD 18.00 N/A 0.00 
VOLUME TOTAL 100.00 94 100.00 
Dale INFANTRY BY MPS 
De OPNPERFORM 
MPS WEIGHT EVAL CUM WT 
1) ACTNSBYMAR 100.00 92 33.16 
2) CMD-CONTRL 58.00 95 ome 
3) FSUPTCOORD 50.00 100 16.58 
SECTION TOTAL 100.00 95 68.97 
2.B STDTACTICS 
MPS WEIGHT EVAL CUM WT 
1) SURFASSLT 100.00 N/A 0.00 
2) HELIASSLT 80.00 N/A 0.00 
3) MVTOCONTCT 32.00 100 4.97 
4) ATTACK 80.00 96 12.41 
5) NIGHTATTCK 28.00 68 4.34 
6) DEFENSE 60.00 93 9.31 
7) RETROGRADE 20.00 N/A 0.00 
8)  TACP 75.00 N/A 0.00 
SECTION TOTAL 45.00 92 31.03 
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ae SPECIALOPS N/A 


Zi) EXTERSUPTD N/A 

Z.0 INFANTRY BY TASK 
Zeek OPNPERFORM 

eA. 1 ACTNSBYMAR 

TASK WEIGHT EVAL CUM WT 
ieee DiscCiPLINE Ee OR0o 100 SoG 
Poe DISPERSION Sao © 100 Graver: 
By) COVER 100.00 100 Sno4 
4) CAMOUFLAGE 82,010 100 29S 
2) 6CU SECURITY HUI 0 100 3.64 
6) RECONPATROL 64.00 34 re 
Py COMBATETRL Se OU 100 2200 
oye BuicTRONIC 52200 43 Creo 
oe NBC 25100 N/A 0.00 
1G) CHEMICAL S00 EKG, ae) 
li) ENEMYAIR p20 0 100 2.98 
iz) + POWS er OO) 80 Zo 
i) CASUALTIES 45.00 80 1.64 

MPS TOTAL 100.00 D2 Soe Eo 
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Be 0 INFANTRY BY REQUIREMENT 


on OPNPERFORM 
Pan. ACTNSBYMAR 
a. 1.1 DISCIPLINE 
REQUIREMENT WEIGHT EVAL CUM WT 
1) SELF 100.00 100 0.53 
2) WPNS 80.00 100 0.42 
3) MAINT 80.00 100 0.42 
4) FIRE 100.00 100 0.53 
5) SUPPLY 50.00 N/A 0.00 
6) COMM 100.00 100 0.53 
7) NOISE 90.00 100 O47 
a) «LIGHT 90.00 100 0.47 
9) HYGENIC 50.00 100 0.26 
TASK TOTAL 100.00 100 3.64 
Pen. 1.2 " DISPERSION 
REQUIREMENT WEIGHT EVAL CUM WT 
1) INDIVIDUAL 100.00 100 0.82 
2) UNIT 80.00 100 0.66 
3) LEADERS 100.00 100 0.82 
4) MATERIEL 48.00 100 0.39 
5) WEAPONS 76.00 100 0.62 


TASK TOTAL oles 10) 100 Break 
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APPENDIX B 


SUMMARY OF UNSOLICITED COMMENTS 


Because of the technique employed of encouraging expansion 
and discussion on each question, numerous ideas were brought 
forth. Although these were not specifically evaluated, they 
are of potential importance and should be at least considered 
by anyone interested in MCCRES. The comments identified herein 
were selected based on their logic and potential impact on 
MCCRES. No attempt was made to support or disprove these 
remarks; such an analysis is, therefore, left for future 


research. 


fee PRESENT APPROACH TO MCCRES 

Few if any units fail. To several respondents the MCCRES 
credibility is in serious doubt because all units seem to 
pass with very similar overall scores. As a result of this, 
the final MCCRES grade is not very meaningful, especially with 
no other grades with which to compare the score. As a result 
of this, two of those questioned indicated they saw no value 
at all to overall unit grade. Consideration should be given 
to doing away with the unit grade and just providing comparative- 
trend analysis scores on a generic basis. This would do away 
with the informal comparison of scores that now takes place and 
provide better support of the intended purpose of not using the 


MCCRES to compare the overall readiness of units evaluated. 
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Bee lwo: MCCRES‘S 

The general feeling among all the respondents is that 
MCCRES is a superb opportunity for training with more assets 
allocated to the evaluated unit than for most other exericses. 
Some suggest a MCCRES should be conducted at both the beginnings 
and end of a training cycle; to evaluate improvements and to 
measure the unit against itself. This would allow evaluators 
to be more critical, because the unit would identify strengths 
and weaknesses early and adjust training objectives accordingly. 
Although it is a costly suggestion, it has potential to further 


improve readiness. 


C. ANALYZE INDIVIDUALS 

Hie weeomnanid meeds to Know not only strengths and weak- 
nesses of unit/subunit performance, but the evaluated unit 
commander also needs an independent assessment of individual 
leadership strengths and weaknesses. Providing MPS's that 
require individual evaluation of leaders at all levels would 
assist the unit commanders to reinforce his own observations 


and to be more aware of where he needs to place emphasis. 


eee CACTICAL STERILITY 

MCCRES rewards those who follow doctrine explicitly and 
avoid innovation. In maneuver warfare innovation and flexibility 
are the key to success. Yet MCCRES measures performance by 
comparison with standard doctrine. Problems here arise not 


Sue Of the MCCRES procedure, but the fact that much of the 
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doctrine upon which MCCRES is based is rapidly being outdated. 
To have an effective/believable evaluation, the whole cycle 
of doctrine and inclusion of new doctrines in MCCRES must be 
maintained. The general thought 1s MCCRES rewards those who 
"play the game" and may punish those who are "ahead of written 
doctrine". Some respondents said sufficient strides have been 
made to overcome this in certain suits, but others said this 


remains a widespread system problem. 


Pee GE VEL OF APPLICATION 

Although Reference 1 states that the Fleet Marine Force 
Commanding General is responsible for application of MCCRES, 
the authority is delegated to various levels. Some of the 
control is held tightly at the division/wing levels; others 
delegate the actual evaluation to regiment/group level. These 
differing procedures greatly change the standardization of 


evaluation and reduce the pool of potential evaluators. 


F. GROUPING AND TIMING FOR EVALUATION 

Most every MCCRES has a different grouping of unit types. 
For instance some organizations evaluate battalion-landing 
team units with the attached artillery, engineers, tanks, 
etc., all evaluated using the MCCRES. Others evaluate only 
the infantry battalion; while some organizations use the 
MCCRES to evaluate artillery as battalions and others do not. 
The Marine Corps standard is that each unit be evaluated using 


MCCRES every two years, to ensure that all units in fact are 


iS 





evaluated. A detailed scheduling system should be incor- 


porated to ensure that no units are missed in the process. 


133 


ne): . 


Ue. 


LIST OF REFERENCES 


Maewner corms Order 3001.2, Vol. I, 9 December, 1977. 


Parrow, Robert H., General USMC, "Congress and the Corps 
CMC FY-84 Posture Statement," Marine Corps Gazette, 
vole of, NO. 4, p. 38-48, April 1983. 


George Washington University Program in Logistics, 
Technical Paper Serial T-427, Analysis of MCCRES Data, 
by Zeev Barzily, 5 August 1980. 


George Washington University Program in Logistics, 
Technical Paper Serial T-430, Assessing Marine Corps 
Readiness, by Zeev Barzily, Lt. Col. Paul R. Catalone, 
USMC, and W.H. Marlow, 26 September 1980. 


Wheeler, Goerge M., Capt. USMC, Evaluator Bias in the 
Marine Corps Combat Readiness Evaluation System (MCCRES) 
Its Identification and Control, Master's Thesis, Naval 
Pecraracuate School, Monterey, CA, June 1983. 


Rothwell, R.B., Lt. Col., USMC, "MCCRES From a Battalion 
Commander's Perspective," Marine Corps Gazette, Vol. 63, 
Mon. 26-46, July 1979. 


Musson, Thomas A., LTC, USAF, "Readiness Measurement 
Reporting System," Research Report, No. 429, Air War 
College, Air University, August 15, 1978. 


Weinberger, Caspar W., "Report of the Secretary of Defense 
to the Congress on the FY 1983 Budget, FY 1984 Authoriza- 
tion Request and FY 1983-1987 Defense Programs," DoD, 
Washington, GPO, February 8, 1982. 


Calucci, Frank C., “Readiness Report," Memorandum For the 
Pimecton, JOInt Staff, 14 August 1981. 


Barzily, Zeev, W.H. Marlow, and S. Zacks, "Survey of Ap- 


Puodemes tO Readiness,” Naval Research Quarterly, Vol. 26, 
oem Dns 29, March 1979); 


McFann-Gray and Assoc. Inc., Technical Report MGA-3980- 


Paogmeaoks eelcoweuny, COmbac Unit Effectiveness: The 
state of the Art and a Conceptual Model, by Steven L. 


Funk, March 1983. 


134 





eZ. 


1 oe 


a. 


iS 


0). 


ey 


ING). 


eo 


Zi): 


Zn. 


oe. 


ZS. 


Zc. 


Shishko and Robert M. Paulson, “Relating Resources to the 
Readiness and Sustainability of Combined Arms Units," 
Rand Corporation R-2769 MRAL, December 1981. 


Zena@Een,p; wom Cc. col, USMC (Ret), “Marine Corps Train- 
ing; An Interview with B. Gen. A.A. Sardo," Marine Corps 
Gazette, Vol. 65, No. 12, p. 51-55, December 1981. 


Veworem, Websters New Collegiate Dictionary, G. & C., 
Merriam Co., 1959. 


Webber, Ross A., Management, Richard D. Irwin, Inc., 1979. 


Anthony, Robert N. and Regina E. Hertzlinger, Management 
Sonenvol in NOn=prorit Organizations, Richard D. Irwin, 
iae., L9OSO. 


CACI Inc. Final Technical Report, ARPS. No. 3379. Measure- 
ment of Unit Effectiveness in Marine Corps Infantrv 
Battalions, by Dr. Richard E. Hayes et al, October 3l, 1977. 


Anthony, Robert N., Planning and Control Systems; A 
BeamewOn< -On Analysis, Division Of Research, Graduate 


School of Business Administration, Harvard University, 
Boston, 1965. 

Lebas, Michel, "Toward a Theory of Management Control: 
Organizational, Informal Economics, and Behavioral Ap- 
proaches," Paper, Department Controle de Gestion Centre 
d'Enseignment Superieur des Affaires, Jouy en Josas, 
BPoance, 1980. 


Ansari, Shahid L., "An Integrated Approach to Control 
System Design," Accounting, Organizations, and Society, 
Vole NG. 2, OD. LOI=-112, 1977. 


Hopwood, Anthony G., “An Empirical Study of the Role of 
Accounting Data in Performance Evaluation," Journal of 


Boece kheseanchn, —iInparical Studies, p. 156-176, 1372. 


San Miguel, Joseph, G., "The Behavioral Science and Con- 
cepts and Standards for Management Planning and Control, 


TEE Organizations, and Society, Vol. 2, No. 2, 
melgv=186, 1977. 


Kohler, Heinz, Microeconomics; Theory and Application; 
Se6OLt, Foresman and Company, 1982. 


Stufflebeam, D.L., Foley, W.L., Gephart, W.J., Guba, E.G., 
Hammond, R.L., Merriman, H.O., and Provus, M.M., Educational 
Evaluation and Decision Making, F.E. Peacock Publishers, 
ace 197 1.. 


35 





Zs 


26. 


es 


28. 


Zo. 


S10). 


ook. 


Be. 


BS . 


34. 


Di. 


oO . 


Bi. 


56 


Bo. 


Tracy, R.W., Evaluating Training and Development Systems, 


American Management Association, 1968. 


Suchman, Edward A. Ph.D., Evaluation Research, Russell 
Sage Foundation, New York, 1967. 


Drucker, P.F., The Practice of Management, Harper and 
Brothers Publishers, 1954. 


Beer, Richard and Ruh, Robert A., "Employee Growth Through 
Performance Management," Harvard Business Review, 
wevyepugust, 1976, p. 59-66. 


Bccay,enoacney H., “MBO Goes to Work in the Public Sector," 
Harvard Business Review, Vol. 51, No. 2, March-April, 
(eyo Dp. 65—74. 


Yoder, Dale and Staudohar, Paul D., Personnel Management 
and Industrial Relations, Prentice-Hall, Inc., 1982. 


Clarke, James M., Major, USMC, "We Need Better Performance 
Soueco ling, ani ne comps Gazette, gune 1982, p. 47. 


Dornbusch, Sanford M. and Scott, Richard, Evaluation and 


Ehemoneoocise Or Authority, Jossey-Bass Publishers, 1977, 
Bem o4 . 


Anderson, Scarvia B. and Ball, Samuel, The Profession and 


mee cehcCe Oneerogram Evaluation, Jossey-Bass Fublishers, 
oy. S.. 


Vetss,eeerot Ha, Evaluating Action Programs: Readings 


in Sccei1al Action and Education, Allyn and Bacon, Inc., 
fo 74... 


Oberg, Winton, "Making Performance Appraisal Relevant," 
Harvard Business Review, January-February, 1972, Vol. 
wee. Lt, p. 61-67. 


Levinson, Harry, "Thinking Ahead: Appraisal of What 
Performance," Harvard Business Review, Vol. 54, No. 4, 
puby-August, 1976, p. 30-41. 


Howara, ).D., Ut. Colonel, USA, “ARTEP,? Briefing by 
Cavey en inteantry Division, July 20, 1982. 


Department of the Army, Army Training and Evaluation 
Enogmam, NO. 7-15, DOA, Wasiiamgton,; D.C., June 18, 1979. 


Sassone, Peter G., "The Economic Evaluation of Military 


Training Programs," Defense Management Journal, Vol. 18, 
No. 4, Fourth Quarter, 1982, p. 39-42. 


136 





40. 


41. 


a2. 


43. 


44, 


45. 


46. 


TG 


48. 


Lawler, Edward E. III, Pay and Organizational Effectiveness: 


A Psychological View, McGraw-Hill Book Company, 1971. 


Beet yworati, Gybermeeies and Management, John Wiley and 
Sons, Inc., 1964. 


Kast, Fremont E. and Rosenwerg, James E., "General Sys- 
tems Theory: Applications for Operational Management,” 


Academy of Management Journal, Vol. 15, No. 4, December 
1972, p. 447-468. 


Flamholtz, Eric, "Organizational Control Systems as a 
Management Tool," California Management Review, Vol. 
PaeioeN@, 2, Winter L9WwEy, pop. 50-59. 


Horngren, Charles T. Ph.D., CPA, Introduction to Manage- 
ment Accounting, Prentice-Hall Inc., 198l. 


Festinger, Leon and Katz, Danie, Research Methods in 
Behavioral Sciences, Holt, Rinehart, and Winston, 1966. 


Berelson, Bernard, Content Analvsis in Communication 


Research, American Book-StratfordPress, Inc., New York, 
US 52. 


Gerbner, George, Holst, Ole R., Krippendorf, Haus, 
Paisley, William J. and Stone, J. Phillip, The Analysis 
of Communications Content, John Wiley and Sons, Inc., 
New York, 1969. 


Druzback, JOonnm, Ph.D., Interview, Senior Technical Advisor 


to Commanding General, Marine Corps Development and 
Bowledi ton Center, August 29, 1983. 


ed. 





Virals SUL LON List 


No. 


Defense Technical Information Center 
Cameron Station 
Alexandria, Virginia 22314 


Library, Code 0142 
Naval Postgraduate School 
Monterey, California 93943 


Assistant Professor Kenneth J. Euske, Code 54EE 
Department of Administrative Sciences 

Naval Postgraduate School 

Monterey, California 93943 


Lieutenant Colonel Joseph F. Mullane, USMC 
Code 0309 

Marine Corps Representative 

Naval Postgraduate School 

Monterey, California 93943 


Department Chairman, Code 54 
Department of Administrative Sciences 
Naval Postgraduate School 

Monterey, California 93943 


Assistant Professor Daniel Boger, Code 54Bk 
Department of Administrative Sciences 

Naval Postgraduate School 

Monterey, California 93943 


Commandant of the Marine Corps (Code POR) 
Headquarters Marine Corps 
Wasaengton, D.C. 20380 


Major Steven L. Funk, USA 
Department of Military Science 
San Francisco State University 
San Francisco, California 94132 


Major Thomas P. Finnerty, USMC 


5400 Edwards Mill Road 
Raleigh, North Carolina 27612 


eet 


Copies 


2 


f~ 














ore OO 
Myce OL 


Thesis 
F44626 Finnerty 
Ge il. An analysis of the 


feedback provided from 
the Marine Corps Combat 
Readiness Evaluation 
pigou cites 
















































































































































~ SO ache « - ale Path ae ee ee e 
—. bay 4 oe ne. teeth Mellen atti 2 lt he ited 1 het etd te ok af ed Atel sh led uk ko iene Sah Ai 2 ap hoalh gs poll cl Neod gist = 
r ‘ ; a oe a wi - we = a <jetlctaadiee  a i I Ch aa fio lg ow el ctl te ad a al ai ed ane ET ae de td Ad g Senet hapa ‘ 
> = ale 5 a han hb hoe ’ Ye r ae wt a. e i b ty we Ew Cd J , i de A ad ° r ra 7 cd * ° ies of P cies 
ae on ee ee BB + PD H ' ef * ‘ - rp Pr 7 ; . a ¢ staat 5 
a sf > . n . r S , 
‘ 
° ; Fs 
F A ' : 
: cs ° Pog ; 
° ® 
Ca r - * or 
FS ; . 
5 } 
; : . 
. A ; . | | 
2 ee . ° 
i : - ad & . 
= Ld a « P 
; . 
a Ca = . = : ; 
Ld ‘ P 4 r 
° 2 ad ° 7 od . J a . i = 
= 2 cl . . = ° . 
° e 7 , 
° - 
e ° " 1 , ; ; ; ; 
ai es Fal my = Bs 
y 7 . ; 
: i cs. 
- a , 
a 5 = ° r Es p = 
# 
. : ; 
“ co 7 
° ° nee - P 
~ . : ; 
P ce A 
A ar) ‘ S “ . St 
* : ; 
= A — 
Pr e F o ‘ . aha ; ; 
ae > - - - : 
bs id ° Cie r ; 
. S ° 
es as 4 . a 
S no - - oy Pa 2 
- Aha P ‘ ro : e 7 Pa “4 ° 
rf ra " ~ ¢ ad al 
: ; ; : Ca | cs Ps 
a 7 ~ é aa ° ¢ * » - 
- “ * ° S ” . “- i e Pa ra 
: one . ? A a ros - - ae 
P F; Py . re - Pe = a ELA : 
: : ae 
4 s ca o = — 
-° e- « . a P a wala 
ie ee . 5 ? ry ae ie ad 
. ° eI ” ° . = -! sl a r P P ere 
oe 7 << P F 2. Si a ’ eS ra rae . - P 
‘ ; d v = < be 4 id b: a ° @ ® ra 
oo D r ° ‘ 7 e ie k ; ’ . ; 
. “e : 2 , - os 
. : i ~ me Pa o. * a = 
er . ry Fe e P 5 we ee o i 2 hid 2 
, a a y Hag < bad ” a 5 ry - - 
r i  & “= a = "eo @ . - = 
- ae . be » cad E . a P 
a - _ » df i , an p Sere i ; 
on ' ~ 7 oe - - = al “ . 
. " Sor os , . 
-. 2 Le 7 ad . * . ’ ad i 
“as : ie Ke : i s at} L on bd ° : ‘se a 
a F . a t ss - 
. P . 2 . : é 
. az) o ms t 
ss ay 3 ie 
, 7 ro = Pad ew 5 a e« 
a tad - -“ he ee e ~ . Cad -_ 
= ad Sd “ ne ' 
. a n é S S 5 7 - ce a 
ve - - e " ‘ . " 7 | 
7 ry * eh % ‘. rn % 7) , Ps + 
Y ¥ ry ¥ * S . Sear ® a . ‘ ° F F ae : é 
a Ps Py a . ad we - 7 « - r ¥ “ f ‘ , 7 
<a ad Ae - ai bs ti . ca , 
¢ ) k Py = ay . 2 i ry . Vi i " 
F * a Si t - = 
og es , = on 
> " Se od * a es bi * ar 
= mres 8 oe re ar ; 
] @® rs . - my " . 
co ei a i 2 ] - er Y ] 4 5 * S 3 
5 ¢ »] —_— ~ as a a rk te s 2 r ‘ ao ‘ ine aH oom 
Po . . . 
oe ey . = 
bd C oc Ce | * Sd . ets 0 
>< 7 . " a am ¥ % é Ce P 7 = " 4 
= N ted . a ' * So be et ed , "4 c ad 7 Lo “ - ‘ 
: y ie | a er r “ Ld L he Pa r Py 
Pe ht ot * q a . wey x sa) Bad 4 * Ds os 
© s Pe ar “ a ¥ 3} 4 ae | % my % . r ee a A 7 
- ar * eh Pe i Yaa ry 3 . a ‘ 
- ‘ nT 7 ™ my . ee ’* x a) * “ . C " r Ps . . e pS 
= > Cat . . . 4 a } é ; 
CO Li al ba vey ¥ lied | acy a L tal ee ’ “ 
= a Pie ee oy he] CT ¥ Oo . - & 7 4 
- CO =] ¥ | t . sf a . fs a Pere ] * A] ~ —P ~ : ke 
t ~ « ‘ a ow 7 a. | = Cd + be ad 1! ¥ bd LU ij , a ar} 
ae ( 3 “es © ~*® LS ae eee he = Ce] ae PY St ee . b i i % Uy a Aha , el te oa iO 
date | ad " a . oe Pre Pe . L a ad . ¥ . - rs = 
fon = ~ wow wen eyes WARE Ieee eee aes ey Ee ee ot ain F ell . a es me 
Q "$"e bt ba a eg aK P a] ‘ € 7 s : e : : 
r Ps a  \ - vw “.% er e "a a} a a Py we Pea " 4 , 7 Ps - 
Cc ) - ott halal i ae " “ a 1 we Oh ae C . . ‘ - 7 e E = - 5 
° Se a. lh ‘ Se * ad : cla) a a ah cr Pr ¥ —" Ps ec ‘ 
bat 5 »* ¥ ~ 4 a a 7 r oh ny a an: . rr 7 F; . ne p ~ i : 
. “aa % we Ss | q Ms a : H sf A ee aa ‘ . i j E 
ad ol ale la t v o% \ . nL aaa Pa] t rn 
be ap pA ea ea EN c : By Ken cs cts Ln Pd : aly is lin by ie J . ead . ed r 1 
‘ a ee Es i ha ae! i i. ie a ay wt Ne * " 4 , c “a a 
._ a a ) + “* i . WwW Bn a ny * hd he ae b os aa z . Li D i i a 
ve * ler | Sa he a ere’, CF eT A + fhe ws ov WwW z ee ie ts we Ww r a t * oi why - ‘ - ; 
bk he oY a th | ied | ~ . a Ps] a i Se . Bk ee Oe s ae | r 7 dt Pa Pi a : od a ~ ‘ . a f D é ees wg Ow eo .- “9 ving r ow ot a o et gfe d 
. Aah Veen as Clipe ae aerate ee, WSC py ee bs ‘ mg 7 eos. * w ‘ 1 A nlite od , - soit “ea " pAhapt a ans . ee eee a Pet a ae al aie 
= , —. oo —s ~ teh ~& ort 2 a a = - oe ct s ‘ A ‘ ie: be ae alll i 7 t 1 mel er golf a oD el - Pe ed te ae The ns Pak od Fad ol i od ad oe oe Pe” atx aT 
so. tir Oe ou ae Rh eee = on ey a ee on ie we PEN a ha o oo aan ne A a , PIP. e ; , Ge ; stad ge pee 
ee ea es Eh ee ae ot A a Pll sel dy ah atte dil bien hy : bon bi i F “ sae Sia Acetate ied ciel thane aed - 
“ Mah A tel tel Cielito hs he ee eS ol ee oe es Mae tae ee ae UP 4 a . Pe a = a j , wi as =o , . on Pit etie re 
an bs Bh oe he ‘he Mh do de eed Toh Be a fhe my Th ts on ch, Oe ste eed ie I ae | . . . F : ron.) 5 F r co " T owW=rvevs a al te AD on at oe - al al ahd oad 2 Pe ory . 
be ee teeth attt ens ali Wika me ghee Wee ‘ be . aa . ! ¢ es * 2 Fre yp eee ae lain dt-detintes theta af thie of ee 
en bei a ha . Pe ek hd ee | es i ee » 2 A 4 E nA LU a ¢ : iat! pols 2 me othe sath ciliated belt seettincnt EN deal at tee el ae el oP = 
Seite Deh het teh Se eh eh i Oe ee tek th th th Hie a) bh tia et td wy * ae yg Sa & a) ’ a 7 Ca - nd ete a ee oad Pe ath te ad at md in as 
ers eT a dees Be i lead a tte So eee ee th Sh a id a Ls pay OT la - latin te 3 “ i wre “—e" bind Aletta I a ¥ etna Girnadcthaelt Aietntari idl eal 1 khan at 01 eel gs, ted erent eee . P 
‘aps Lo Best Le ap Raita Bk ee amr hdale ED he od dine n, Heo ek ly ate bi *“ . . 5 f hae aw Tr ay pe nae. ern) eae da de ha wr saith dha od ehhh dele oaks any page See 
: ae Nocti nth ata are pty AIS wh aa : - | ae 7? ** Gay werek . . grrr ee te . hs ttl alan | A I ed "> v hice Oe ae Cs alate ial ol hid arciaddl at tt | Pia 
Ox tee bn Se ah SE dae Td toe Rati tee Vite te te he ye wes "e &F ‘ adel - Ls 2 tt Lee eat . —v eh eee ee eas : o i ee | ta he a Y nl lhiden hes Ot tha a a ted didn, Matt melauee ite oe ee, he te ee ee en 
Shire eee NP Rott th lhe i A ee a Rt ‘ Bgl t i cane a ay, Pe ah ng latthel ail Lo il Deel ot, ake a a onda aha, hgh tit dh date nb ok et tat ae 
pS iden lp Aeety OLD att Be ted th on LAG Ie Daas edie hte ‘tT Dinh Me bee Aelia Wha a h-hh v G Z a t . nh ee D To ¥ awe a ae PTF CN, St i leet ie tt od fo aad a pals ial ded aie F 
ee ee a. Me de eee u% De ee Bs Ge 2 baht ee a Se elite ah Gath i we ba ee . LL ¥ u ee ¥ . . td ’ f peal sad Ve PPV wh ee UF Fg pw “at ol ade wee ieee dated Fd i ed 
hated ats Ah ate ain al hh ba es he 5 i te te hah he ke ah ee bedi a dy eeyer sed wits ah Elle etal ¥ i J iw se Oi w a . eT FN pega FO oe He a PM itt Ad en a Pe 
ae en ne he ie ee et ote en Tee ee y " Pe Ns dies | La SL t bi hs Oe ee ler ee mee rT ; vor, “4 ed ee a] ery gs: 09 see Ul adel a ae ek hot ctok dal of de tae 
bb Bh Ran yinte inlet h-hh Dadi bane ad tt te eM hah ce a ne he ke Uk me I oe Or ees si we c “ r y oe a ye, .ee Pay — Ny Pact te aah A dh death Bit de ae at ht Pes 
tyes © ee RAN sy eee te ee wi on se Pah oY ’ ‘ se be pn on ih wee aad i 8 hte 4 r Aa Oe a oe a eel hi Al heh lalla taal ot MA thal te ot ok he dt To he Pe See 
ota th ete ete elo ke ae ee Me a ro Ah ax to alah ee ett ah te eid Le | ach stot, | ' . r vs = B 1D. asada 09 ado had aig - Lp ietieatilitAD com Nh lat itll LP Jeet ol Atl afl ti telin taal eae i dete, ah od, at nt ad a oe 
tsb lap tanta ohion,: Ky dylan ds Verge ho Noe ph tebe ada thee . ype So oh “ine Si Partie eatin they hy cw J 6 ait vata lit el el a et et ae 3 o , >” salen di fh beh le Sok all ato he ed oa a ea tat cath de tex ah Mn nano ot ak edt aden a med ot af ant de al aha ae aa 
este, Se tndte th it runt tep Nnde Dec Stele Baths thee hee Mee Le te ect toe he Abia: Lal LL aa Ce 7 prey oa . v aoe tact Ph r — 7 ph fh ha lala a spd Adee alah ee ae EE ie Pe Tee Le 
: Satepiel i eager Seana Se Ea he Res i laa Aap adi cal Pee AGL ri 4 Ahi ha eile hee S 7 Ya - ¥ « \ re ? P Tere as alae ee Atti atthe titel debts d de nat ednsied fakin, Lt kate ok re Lchathanikde .4 redigla Adder te oe 
Seclpctnat Dip lp eth. otha Ka, eh 5h Wh ahi teed et Uh oh Ser Me a Yok ae en ee oh te Mh "fie " a, wwe’ ids li ls ~t dip tected pilot De ipa ¥ * Fa ry ay OF CTE, «rrr wre hdl a4 tes ate ah ick Aa oll atthe dnt nde hon at a, shat inh 2 leat that Ladd he ee ee 
Seite k bitte Sete td oe tity the okt Me a hs Ch va od thee ee ee th nt heh eh te La LA aia “ fe: Sot “ od ‘ - Wikcdeni ith ins & OD Ue dee hes A Mal ME La abe ae wt Pe ot een dah nl glider xuth act thadel a ete Ea ge El 
ee Oe er eo he a tale, hee OS PLR The ee ee et Al le M es atte a ? co lei De iat aie de Dee te Lo) Ve es ores data a hte falta diet Hind hake o att eoldae re ee 
Var A ee fray oy ppc eT iy Debbi th fy LA ae Ar “iat hee ea y tit © s “s ich b ive I ” ] i a Ss i ee de i a La Se tl Pe es | bel alle el a i eet AE Ded ret ad dt a Da at A a Oe oO a 5” a8 eae 
CARTE SESE GRRL IG, AW YE RA PRE OPN TE EIR Pht A, ol eH id ae ve vs LSiek a a a el a vw  F. c wet ro s f oY ‘ ww err = at he A ae ot ae . wd lll ll ons lie el oo ee alat phat saul oa. owe giana 2 0-9 
et et ee kee ey ierwomme & O elarn wae hie te te tee + ee ee @ -¢ & @ way'e® : Ss o ¥ « ie | ei ee ov, “ or bite dl bd an we iia tl Montel ned TS, ae. ae hs te i alt ae! i, oe ha pK etl ta bid at 9 ated dh ad md aes cet edad Mates 
eee p eliotn elites TR) tae GL pera Ral eth Pe, neds hie herve th Nts cei cote! i allie dl all A ie | or 4 ¥ pore oi , j rn Ber " / Pe it aed oF ek a dk or a i ee Sle ak a a oes  peammel ot ad’ tal a 
Mei, Tepttele ce th Roe etd atten the Satie, Ha bette httnstilin Mii Nia eh den ok 2h Saha hh ie ea i tre * = * Sel ae} | 1 Fy A ‘ S m4 I~ ee oe 0 aoe ere Www nll ncthnel th the th hl tcnids WS dk nh cnt taba Dt Dee Sn, ed 
; fertile A i ht Nig Se hay Atel Rind Vitette Wh Ng Mh te ok hte PU ee ha Me shee on Oh ok PR teh eh Ein nee et ka) LE. tec Gate a U - U Y RT ss etten die nhl alae ak ne A "endl ate ato (mitted ok i dal eo Tio ee Pe ne et ele hel nee eee 
bedi liad Ssats tuhadat BA LA oath te Lb te thal Se thet tee th eee oe tA EGE a a et en th ee ‘sg \ id we oft hy Ld Li Tein i ~ lpia ct Iola | : a rly heF poneen i el, La la nv Xitel ak bias a Aeon Ye. satledl wh th Be the Bn) od Dh La a 
by Aipthte Sp So he Se cbse Ves he Mah. iA Te st rhe pha scot al Ee Be Lg wi ad i tl ee ey were de pei a — : ra : att Mids JA Did os cll de del ated cnet Mths ated 0 bo cae ar ce ihe t cath chihane All d alot Le 
beithae th. Oh Whi Abeta’ tied Nan Aan Si led) th Sah dk i hla et ble ee ae Be | F : f : C — Ae Ay V8 " i ls “ea he tne da ae ee aero e0 | te det eat an ah a hi ted ide intel Milde adhd sti tl ohn aaa aa Tk. eee FT ee ee 
teh “ab toe ahs Lond hk ek pt alle AN it Ie, ciel Nan aha hia Be ye : Aes Aiea ett 4 he Ay ge! : ‘ Ala de ii ett i MP MN Sell SLs Seat oid dee La LA 5 Daag tga Peheat elctih- dln. ckecehde Alber a a ee ane 
BWM cA ee ee Be | ash 0 ie ete ALA ie Th eine oe aL ; : eh i as ae : t o ade) re ee ees ' P S 7s dil nai lielinadln Maciek fe nto oT aati Lyngdal ee eal rh cere ataok ioets 
es er a a ees Yi he ee te i ae | Mae Bk he Bieed Y ae ian ae Tal Dot o. Tei t E bidet p= . bs ¥ , hi my i v J Latte adr Tt RP Fo wren ie belle nh ei del ileal Mod ooh We ee a i Dali hth ieee dl hate at eek 
righ & were ah pee At ee a Wwongey cnt. he o i 7 rs \ Me ee ie So ee | A eS a od 7 ves . : " be ta 4 7 ¢ wy, - Ve 42 Se ? i el eal 1. Laat a Po fo ae Se et Ak i al id del Fy at eh of het aed od a a tT ie den dena’ Lh eke wa 
teen) oer eer te ya wt Cries Tete pila at Siar Se Sat i ’ s eS Me : HA , -" d ¢ . Siete dad ~~? ory 7 Sl ed ial lt al Vi ide tte tt ae he ae edd ol on ad Oe ee 
a a e- ea ee eel ob ae rie eae ie ee te | . . i ea et toe ‘ ae é da v _ a, « r ied | o ‘ 4 Dy 0. @9. rs a Ooo ral Di ory ’ ae daa PP i al Owe oka v aa ot Pe ee a ea fo -e 
pefal ip asta Nh Ae enh ka Rett ah ae Oh Sh an ik ies Whaat ee he he niece dtatae i Th oa hoe ‘i oS a ey ae - . cag a Piet ao OLE allies Aides coliat dd ed it alt ns ee ae Watlitill giddol x bien nnd ee Ftd host etd oa ae 
ee ee es Tt et debs “an Yo, ep Ie bie, oe, sh hh A he os + a b ak ee Tl tl i ae Ok ae bie Dy (swe bh ht llhdad be Ce a yt A eile! Lh mele Tie Sonn Or PPE I FOP VR elite ee hs ak ate A tee del Ot ed dite dik a at ole eid eb ai et ee Plea 
ie Lal Meet dod Sh ied) tad hl brie etal ee tet ok hh ki dial hath Pile ited aad Lot Bes Ye cM lite. Bde te et a Rg Mets Bh at + Male Naat a 5 c > "eh ie ea el compile Had  arteseeth Sik lan tia tet 5 edd jad Mths Lhdath ah Lok hal nas a ee a eee 
SOE e end 2.5 ore Ss ns th Te oe eo he eb% a "he ta a « Yi lida it te 3 Vv 7 ba By =o i ‘ oa | rt Deeded 2 Ys Ad inthe. ell al’ ok call dak a A ade te ee a pd ae ee Le eae Barhat os pa peg fanapeae 
ial ich ai! y ee hee eh ie Cae ot in mee Rad dnkn hee teh be oie el ak eo ww ae : a be . Ca ere aire: . : Fated fy wea p ee Ne ee that aed seca Nath ae oa LER TR  P AF he Sak da pled orate pe 
sl ts Vinten A. Dahan, hah Oh Sh AN a bee ier) te aT ka oe ee ee eh Sere’ hPa * WY CR - ah oe ope a 7 ae ee a tah Cu " SN Ah Dalal de Bubehbes tia Adel tee te te thie di ey Waa bight conde eh pedis nel dee al at at de al dt ede, ee el re 
ripe 0 pe A et aie bo thay Sata Mea a ote ; Phe A 1. . : e = lh . a hee Te a el ae ae) 8° OO 9 Te mie | et lini Mell all etd So a bt ake | i ah ek ks ed titted A del th 
he tA sie ti Stone: maf a ty Me track eae nee 5 ih pe “ts ps ae b pte phos Eo r > al) i J J =r Moho’ ik Se hei ot ee el ie he ee te er Ld add i eae - ed ad ll. o mee tek of ie an Le ed atl ~ bal te ee thee hat 
PG NT Athi Uriel, dep ik aan. 6, dake fb th thd Tole tla Mod Va, Wh Seta a ech LA Ae Bota tah te Pte eh ee te \ \ head % - ea -! ‘ id dd ahr A aed Ar Sil Sila dtinays Ard aston de arden pts Lad in Ait lS tol te ag ch the al tated, teh aod nl? ae PO 
tah Bho Js Det tas Se te ee ee A he deh Oi he Yo es te eB a te es he at eS 9 a ls a rk bd La a Ai oa. bd in i r To ‘ res ie wee heat tt dei tet be A lt At bh a ee Ca 4 sthedth dented i die dint od at de athedantetn da tie Mo el fared ee teh nel 
ee ha. than be io lke ed te te kt le tu Met ke i ot ie ee ae ay ty Sil, nl 1% ‘ Bae oe es TL $ heel wae eb te Ak es Michal Maal ilar? nds ta od ot a ok eee, ts AE edad i Nd A Beak ale oe are hentah tk A ald apie ke oe ot ant oh acne ee 
Rada ELL Sb elite he Diba, Dh hl lpr, ©Rb-dy 9 aapnept bei Lid ths Tar ait ek te eh Beal ah bad al ae] he ° a ee ae a a i eat, ~ ellie Rai ella, Deche BR et Api inde ea dieind at tad? Pai Sl Neo Michi Whar 1 eta dt Oe et ae vale tthe indie Methdt 6S a dak! cnt! otal Mh th pleted dake ate meee 
pete Napkin te Sadie Sa, Wh clbet hah “shld fies Oh hey ey Mee Oak PS le fi ih Mt Chet ie Se es y | . on , eT 1 = Ye The 7 ie i PaO jus Haintes de Ah Lidell, ot a he eee x Ot Saat fan ee Lied ciel batt anil Ad of deead tlh eae ont teat hte ath i> ade ee 
el yb bene Mi fd ta th Sti Re Nt Ul Seibel ie Wes a RO aa el 3h hh SOR Ny ve PTY eed Ces le ' C Dieta at oe  gterti eh, RA & aes ee a o% hae os tt ae ae, Nae ha ee We a ta Ee oe we iba sate i aiedine itl idea i Lodi A tae la ns ot a dana ee ae ee 
jee lly da a eta hese dan eat Mabe Attia th eh. Rete ee eT ee tke Re a a ta | heii ash The destin Lk. pp Wa ‘ ‘ bh © i a Ae he Th a-ak, aa an LA Se aaa ete ri Dd aah Se ches A CL ae Dhee oat aioe a Like he a ek hr eee bor lla dl te ti hotell d tht he ds Neel al tee de nit at ace tek dala a leet Pe ae 
Rati eh Sintuth tate sith teed Fai ete he ee es oe ee Pet er ere iver dk ah ele me, En 3 e A tech. : [an ¥ ha a ws Patty ia heehel dak ‘ ¥ Chew laa ml * (al I Hh Nk belle ‘a ded Sling Me obbatal heh Ai adi dik Lae bee ly sll de dh nthe Mall ooh A dn al Geli aol A lbchoell AE ft ntl a te ge en 
ih thd) Taal Hk ayer | hd te Sale ah Ph a ah Phat eG ok Pe A eae a : ban eb Peete hy % alae Vapi Aa 2 de c Af a " y ; nes Led ON LKB Eade ion aes a a ee papell tant Sa nat ER Mh hag on PT Pe ee Sr aimudien sae 
bie aah ok bleh tt hate Meee ny eta A em ea Te el a ee eo ark ; al pea deine at a i ae t Fy id ype Mi, te eet ied ae de) Te es et IT) EP LP edhe pias AeA chan tes Sed ad ak te pe Le EL nd Tle SY LOE afte on atl limes ppenconel kee 
Reales tadin Salta eho Th Ait Mlb bt eee rteds- bh Ie te ead , bd as fis tle tL it . ye, Ay bol sa ad lea Si ‘ mi ly ~ a . het’ ak hed i SO AA pln Ri OLA ae colt CA inte a apa Aol od AAS cee ee ed od ot shah nadaetieds The a iain i de ea 
tet ht a ee ee ee ee ar De Bake Vw eet A a 1 Fore Std -* AP. . ‘ F < Sd 7 ¥ “ alah dies at eo rr a) ee ee YEE) BS eee VT per & eth tidied gash ee ee ae 
lessee Mae heh Sih tps Shei co th th teak Pt ce oh Tee SE e “we RTE Fe Le ery : * “abs Y Fai 4 ~ ; Eby uta dn Anthea vy gall aalipedl a met lh gs ie decal Abita bi hh Mth de We ali Prt ted Aa Ol al f ncn ral tte donne Ai gh a tk ad oe od ak ee ee ee 
be tay darth sdb of Kala id eee tN ed / : A lial Din, Se Fe De Hl ite Ay ven e - . nay 4 cS Has . ae * te ~ pe a my ee ahah iia Ld ‘ COPE TY yr Pagers KD FTW OD OC tm Fe oF OH GF ye Ghee ae Our rt « Ohad shed 
Rt Sy Rhea gn Seth Sdn ey bs 2G eh BOR a hepa fk SAY ee ee ea . : j ’ tie Y “* ats et id a ed le a ani so shall tipies is en col Ae ae LAE gi, bo Re 2 oat oh a Ak 5 viehth dd Lik dink ight at hte akal to ee, ane Prot aed 
te es aes os hbo Mirtbet 1 ee a ht be Be gee ee aR, bl ¥ = ty i i] + 7 . Gi de a ea fires «7 yr betel A laced le ol hal tat tek at ate om a bw Kt tee Ae hal ae ae Pg Ee a 
Topepy ae es Leh bah Se a Sa bie ek PR Pe i Mik i TAs hh eld a a a Pe wy a rye , tae et fel Le het ae A aD ¥ Sl die D4 aihactided sal ae nt Red 4 on oe ie ea Cn bd sl bol aah dk td et od At POF oe? FAR PPE F 
dp. tp Sheysteh eee tb aoe Fala ab os ahs BL Le Gate a he Be ad ai tote kh Aa LM Atel on a AC | = bes, 8 Ls i otek fan load op -y i le. mont UN pe nt aE fe 5h) nat bn OEM foal 1d, Meta ee es ttaieh Prd eh nck MN eh ee NEL Sah Pend Se 
"Seeti h Me peel td ie cnt see ee ed ee ee Ws eae a noe YoVi da - ; ton Pins Lr petaa! | Te dm vi eet Pree ie a ital hes ai lah da Le FAT 5M , Lelie a dod is ee a eA gts an ated Se tee ee eatin lee & (Ao ne 
a en et nee tt tee et ee ee Pe ae é uw 8h 9 ee A er 4 a ened Mk det te mit ah ota oot be on Panes Seek Se ee et ae Oe Ce we ee eee BOT 0 ph aap et 
fa pile Mid A te bic Tonk he ae de ‘ yl ~ Seles ar La - tg See Mt taal ion oi hele! aed Pett hed ¥ ad ada att a NE eae gh elbe- a ARG rng cag “AS” pry: 
Se Se Rew PAY KyeD WAM Y whee eee a ye o La AT i ee th tied ey rs Pr oy  arllatl at aoe ke ee A se bee | Ce ae Dy" ow ae ot Se oe en ne att oh A Moa of de) at 
ae 1 an dbtehdek See Soil) ah Dae et ie ie Bde | ik oa eae et, By ; T - nS ss - ret cere Mg re gwre SS re eae ped ental lie ited Aa, DA 8 Cale Ab ge eek ee Oe rfiyt pas paee is eel i a Tot ay 
Po : en merwur wes Cerys. Le Me tee ci ara tte Oh A f tS fe . (i fi i iC e rn 3. 2 - A tk el ale Am ls alt ft, el sie ie el ee oe ee a] Be Ci de i lhe Ske Bed ot _ ae at OF a Pl. aur. wey? ad eit TWN es 
bet i dhe hee ee eR, od oh ek Reet ee eis So aD ee res bith ek : tO Lo ae e r m i F : wee B 0 2§PR Og ee OORT Fo yee ir seen at Mita ht teh Bh Gh thats iene het Lhe Of nee Dia eT (pli dd geez dal ae od nl ot Oe ee 
betel ih tai Sie Bat le ba Sid Sk ho ee A a Ok Oc : x spss ore f 4 ie ce ut eae ny Wied J Sled af tah i lh get ob a ant ahh oe at ae es it atl Med ec nk aes eat ee ate gh Aa ea ente lt didn af Aa ie ee ety 
phan lls Ribs Meld, Bly ih adi de ied dat gy. nw ay ie . eM ahd Ria eh MLL, ‘9 a f x id 3 - : ni ' i boomed 2 Tadeg rt i ¥ 4 l,i ee rrr ae a - > i omy ¥ hate de thle LR Le ol th alt all ol oe nN fF ib gaat sat? odd od dedi ol lt el et eS vr ait ae en 
pene tk eh Loe tak Pa OR he Rk St Peet an 4 i hala UR ah aie L co i Tt ee Te Bren pte Weg rok 1g COR OT 8d eg eey 9° yeas I Poe elie alk Kirt cok ak all os td F : 
bh visti eG DOR Walt he pee eet ee 7? he ie A ah ¥ d t ri Poe ea x a me ~ Biot igh eM Ny alah th th A . ; 
NR ret in NE eR DL open eRe: a yt eee 
a oe oe ke oe ee ee ee BP OL ad S 















